This is a spin-off post about failing attempts to fix the problem with a webcam’s keyboard buttons. Namely, that the a shaky physical connections caused the USB device to go on and off the bus rapidly, and consequently crash X windows. The background story is in this post.

There is really nothing to learn from this post regarding how to accomplish something. The only reason I don’t trash this is that there’s some possibly useful information about udev.

What I tried to do

There is a possibility to ban a USB device from being accessed by Linux, by virtue of the “authorized” attribute. Something like this.

# cd /sys/devices/pci0000:00/0000:00:14.0/usb2/2-5/
# echo 0 > authorized
^C^Z
# echo 1 > authorized
bash: echo: write error: Invalid argument

The ^C^Z after the first command is not a mistake. The first command got stuck for several seconds.

And this can be done with udev rules as well.

But surprisingly enough, there doesn’t seem to be a way to avoid the generation of the /dev/input/event* file without ignoring the USB device completely. It’s possible to delete it early enough, but that doesn’t really help, it turns out.

ATTRS{authorized} can be set to 0 only for the entire USB device. There is no such parameter for a udev event with the “input” subsystem.

Some udev queries

While trying to figure out the ATTRS{authorized} thing, these are my little play-arounds. Nothing really useful here:

$ sudo udevadm monitor --udev --property

I got

UDEV  [5662716.427855] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1 (usb)
ACTION=add
BUSNUM=001
DEVNAME=/dev/bus/usb/001/098
DEVNUM=098
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1
DEVTYPE=usb_device
DRIVER=usb
ID_BUS=usb
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_USB_INTERFACES=:0e0100:0e0200:
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_FROM_DATABASE=GEMBIRD
ID_VENDOR_ID=1908
MAJOR=189
MINOR=97
PRODUCT=1908/2311/100
SEQNUM=24413
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716427506

UDEV  [5662716.430744] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/2/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc02ip00in01
PRODUCT=1908/2311/100
SEQNUM=24420
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430425

UDEV  [5662716.430935] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/1/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc01ip00in00
PRODUCT=1908/2311/100
SEQNUM=24414
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430396

UDEV  [5662716.433265] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/media5 (media)
ACTION=add
DEVNAME=/dev/media5
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/media5
MAJOR=509
MINOR=5
SEQNUM=24416
SUBSYSTEM=media
USEC_INITIALIZED=5662716433110

UDEV  [5662716.435400] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/2/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc02ip00in01
PRODUCT=1908/2311/100
SEQNUM=24421
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430425

UDEV  [5662716.436539] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/video4linux/video0 (video4linux)
ACTION=add
COLORD_DEVICE=1
COLORD_KIND=camera
DEVLINKS=/dev/v4l/by-id/usb-Generic_USB2.0_PC_CAMERA-video-index0 /dev/v4l/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-video-index0
DEVNAME=/dev/video0
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/video4linux/video0
ID_BUS=usb
ID_FOR_SEAT=video4linux-pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_V4L_CAPABILITIES=:capture:
ID_V4L_PRODUCT=USB2.0 PC CAMERA: USB2.0 PC CAM
ID_V4L_VERSION=2
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
MAJOR=81
MINOR=0
SEQNUM=24415
SUBSYSTEM=video4linux
TAGS=:seat:uaccess:
USEC_INITIALIZED=5662716436054

UDEV  [5662716.436956] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121 (input)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121
EV=3
ID_BUS=usb
ID_FOR_SEAT=input-pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
KEY=100000 0 0 0
MODALIAS=input:b0003v1908p2311e0100-e0,1,kD4,ramlsfw
NAME="USB2.0 PC CAMERA: USB2.0 PC CAM"
PHYS="usb-0000:00:14.0-5.2.1/button"
PRODUCT=3/1908/2311/100
PROP=0
SEQNUM=24417
SUBSYSTEM=input
TAGS=:seat:
USEC_INITIALIZED=5662716436500

UDEV  [5662716.591160] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22 (input)
ACTION=add
BACKSPACE=guess
DEVLINKS=/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event /dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00
DEVNAME=/dev/input/event22
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22
ID_BUS=usb
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
LIBINPUT_DEVICE_GROUP=3/1908/2311:usb-0000:00:14.0-5.2
MAJOR=13
MINOR=86
SEQNUM=24418
SUBSYSTEM=input
TAGS=:power-switch:
USEC_INITIALIZED=5662716590816
XKBLAYOUT=us,il
XKBMODEL=pc105
XKBOPTIONS=grp:alt_shift_toggle,grp_led:scroll
XKBVARIANT=,

UDEV  [5662716.593390] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/1/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc01ip00in00
PRODUCT=1908/2311/100
SEQNUM=24419
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430396

UDEV  [5662716.595836] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1 (usb)
ACTION=bind
BUSNUM=001
DEVNAME=/dev/bus/usb/001/098
DEVNUM=098
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1
DEVTYPE=usb_device
DRIVER=usb
ID_BUS=usb
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_USB_INTERFACES=:0e0100:0e0200:
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_FROM_DATABASE=GEMBIRD
ID_VENDOR_ID=1908
MAJOR=189
MINOR=97
PRODUCT=1908/2311/100
SEQNUM=24422
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716427506

So the device I want to avoid was /dev/input/event22 this time. What’s its attributes?

$ sudo udevadm info -a -n /dev/input/event22 

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22':
    KERNEL=="event22"
    SUBSYSTEM=="input"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121':
    KERNELS=="input121"
    SUBSYSTEMS=="input"
    DRIVERS==""
    ATTRS{name}=="USB2.0 PC CAMERA: USB2.0 PC CAM"
    ATTRS{phys}=="usb-0000:00:14.0-5.2.1/button"
    ATTRS{properties}=="0"
    ATTRS{uniq}==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0':
    KERNELS=="1-5.2.1:1.0"
    SUBSYSTEMS=="usb"
    DRIVERS=="uvcvideo"
    ATTRS{authorized}=="1"
    ATTRS{bAlternateSetting}==" 0"
    ATTRS{bInterfaceClass}=="0e"
    ATTRS{bInterfaceNumber}=="00"
    ATTRS{bInterfaceProtocol}=="00"
    ATTRS{bInterfaceSubClass}=="01"
    ATTRS{bNumEndpoints}=="01"
    ATTRS{iad_bFirstInterface}=="00"
    ATTRS{iad_bFunctionClass}=="0e"
    ATTRS{iad_bFunctionProtocol}=="00"
    ATTRS{iad_bFunctionSubClass}=="03"
    ATTRS{iad_bInterfaceCount}=="02"
    ATTRS{interface}=="USB2.0 PC CAMERA"
    ATTRS{supports_autosuspend}=="1"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1':
    KERNELS=="1-5.2.1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="ef"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="02"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="256mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 2"
    ATTRS{bcdDevice}=="0100"
    ATTRS{bmAttributes}=="80"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="98"
    ATTRS{devpath}=="5.2.1"
    ATTRS{idProduct}=="2311"
    ATTRS{idVendor}=="1908"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Generic"
    ATTRS{maxchild}=="0"
    ATTRS{product}=="USB2.0 PC CAMERA"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="16"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2':
    KERNELS=="1-5.2"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="100mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0100"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="75"
    ATTRS{devpath}=="5.2"
    ATTRS{idProduct}=="7250"
    ATTRS{idVendor}=="214b"
    ATTRS{ltm_capable}=="no"
    ATTRS{maxchild}=="4"
    ATTRS{product}=="USB2.0 HUB"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="409"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5':
    KERNELS=="1-5"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="02"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0123"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="73"
    ATTRS{devpath}=="5"
    ATTRS{idProduct}=="5411"
    ATTRS{idVendor}=="0bda"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Generic"
    ATTRS{maxchild}=="4"
    ATTRS{product}=="4-Port USB 2.0 Hub"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="removable"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="69"
    ATTRS{version}==" 2.10"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
    KERNELS=="usb1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{authorized_default}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0415"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="1"
    ATTRS{devpath}=="0"
    ATTRS{idProduct}=="0002"
    ATTRS{idVendor}=="1d6b"
    ATTRS{interface_authorized_default}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Linux 4.15.0-20-generic xhci-hcd"
    ATTRS{maxchild}=="16"
    ATTRS{product}=="xHCI Host Controller"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0000:00:14.0"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="454"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0':
    KERNELS=="0000:00:14.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="xhci_hcd"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x0c0330"
    ATTRS{consistent_dma_mask_bits}=="64"
    ATTRS{d3cold_allowed}=="1"
    ATTRS{dbc}=="disabled"
    ATTRS{device}=="0xa2af"
    ATTRS{dma_mask_bits}=="64"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="33"
    ATTRS{local_cpulist}=="0-11"
    ATTRS{local_cpus}=="0,00000000,00000fff"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="0"
    ATTRS{revision}=="0x00"
    ATTRS{subsystem_device}=="0x5007"
    ATTRS{subsystem_vendor}=="0x1458"
    ATTRS{vendor}=="0x8086"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

And what udev rules are currently in effect for this? Note that this doesn’t require root, and nothing really happens to the system:

$ udevadm test -a add $(udevadm info -q path -n /dev/input/event22)calling: test
version 237
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

Load module index
Parsed configuration file /etc/systemd/network/eth1.link
Skipping empty file: /etc/systemd/network/99-default.link
Created link configuration context.

[ ... reading a lot of files ... ]

rules contain 393216 bytes tokens (32768 * 12 bytes), 39371 bytes strings
25632 strings (220044 bytes), 22252 de-duplicated (184054 bytes), 3381 trie nodes used
GROUP 104 /lib/udev/rules.d/50-udev-default.rules:29
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:8
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:17
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:21
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'input_id' /lib/udev/rules.d/60-input-id.rules:5
capabilities/ev raw kernel attribute: 3
capabilities/abs raw kernel attribute: 0
capabilities/rel raw kernel attribute: 0
capabilities/key raw kernel attribute: 100000 0 0 0
properties raw kernel attribute: 0
test_key: checking bit block 0 for any keys; found=0
test_key: checking bit block 64 for any keys; found=0
test_key: checking bit block 128 for any keys; found=0
test_key: checking bit block 192 for any keys; found=1
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-input-id.rules:6
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'usb_id' /lib/udev/rules.d/60-persistent-input.rules:11
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0: if_class 14 protocol 0
LINK 'input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' /lib/udev/rules.d/60-persistent-input.rules:32
IMPORT builtin 'path_id' /lib/udev/rules.d/60-persistent-input.rules:35
LINK 'input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' /lib/udev/rules.d/60-persistent-input.rules:40
PROGRAM 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22' /lib/udev/rules.d/80-libinput-device-groups.rules:7
starting 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22'
'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22'(out) '3/1908/2311:usb-0000:00:14.0-5.2'
Process 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22' succeeded.
IMPORT builtin 'hwdb' /lib/udev/rules.d/90-libinput-model-quirks.rules:46
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/90-libinput-model-quirks.rules:50
IMPORT builtin 'hwdb' returned non-zero
handling device node '/dev/input/event22', devnum=c13:86, mode=0660, uid=0, gid=104
preserve permissions /dev/input/event22, 020660, uid=0, gid=104
preserve already existing symlink '/dev/char/13:86' to '../input/event22'
found 'c13:86' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:85' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:84' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:83' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
creating link '/dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' to '/dev/input/event22'
preserve already existing symlink '/dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' to '../event22'
found 'c13:86' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:85' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:84' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:83' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
creating link '/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' to '/dev/input/event22'
preserve already existing symlink '/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' to '../event22'
ACTION=add
BACKSPACE=guess
DEVLINKS=/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event /dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00
DEVNAME=/dev/input/event22
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22
ID_BUS=usb
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
LIBINPUT_DEVICE_GROUP=3/1908/2311:usb-0000:00:14.0-5.2
MAJOR=13
MINOR=86
SUBSYSTEM=input
TAGS=:power-switch:
USEC_INITIALIZED=5662716590816
XKBLAYOUT=us,il
XKBMODEL=pc105
XKBOPTIONS=grp:alt_shift_toggle,grp_led:scroll
XKBVARIANT=,
Unload module index
Unloaded link configuration context.

Other failed attempts

I tried the following:

# Rule for disabling bogus keyboard on webcam. It causes X-Windows to
# crash if it goes on and off too much

SUBSYSTEM=="input", ENV{ID_VENDOR_ID}=="1908", ENV{ID_MODEL_ID}=="2311", MODE:="000"
SUBSYSTEM=="input", ATTRS{name}=="USB2.0 PC CAMERA:*", ENV{LIBINPUT_IGNORE_DEVICE}="1"

(the := assignment makes this assignment final).

However none of these two rules managed to stop X from reacting.

Setting the mode to 000 made the device file inaccessible, but yet it was registered. As for the second rule, it doesn’t help, because it indeed set LIBINPUT_IGNORE_DEVICE correctly, but for the wrong udev event. That’s because the udev event that triggers libinput is based upon that the KERNEL attribute is event[0-9]*, which is executed earlier (see 80-libinput-device-groups.rules), but ATTRS{name} isn’t defined for that specific udev event (see output of udevadm info above).

I also tried RUN+=”/bin/rm /dev/input/event%n”, and that indeed removed the device node, but X still reacted, and complained with “libinput: USB2.0 PC CAMERA: USB2.0 PC CAM: Failed to create a device for /dev/input/event28″. Because it was indeed deleted.

But since it appears like X.org accesses keyboards through libinput, maybe use the example for ignoring a device, as given on this page, even though it’s quite similar to what I’ve already attempted?

So I saved this file as /etc/udev/rules.d/79-no-camera-keyboard.rules:

# Make libinput ignore webcam's button as a keyboard. As a result there's
# no event to X-Windows

ACTION=="add|change", KERNEL=="event[0-9]*", \
   ENV{ID_VENDOR_ID}=="1908", \
   ENV{ID_MODEL_ID}=="2311", \
   ENV{LIBINPUT_IGNORE_DEVICE}="1"

And then reload:

# udevadm control --reload

but that didn’t make any apparent difference (I verified that the rule was matched).

And that’s all, folks. Recall that I didn’t promise a happy end.

As I rooted my Google Pixel 6 Pro, there were a few things to get in place on my Linux Mint 19 machine. These are random notes I took as I went along.

Install ADB and fastboot

# apt install android-tools-adb android-tools-fastboot

So that was easy.

Next, opening the phone for ADB access, the standard Android way: Go to the phone settings, into About Phone. Tap seven times on Build Number. The phone prompts for the PIN number, and then the “You are now a developer” message appears.

Now, under System, there’s the Developer options. Enable USB debugging. And then at shell prompt:

$ adb devices
List of devices attached
23011xxxxxxxx	no permissions (user in plugdev group; are your udev rules wrong?); see [http://developer.android.com/tools/device.html]

Arghh. This was because of a lacking udev rule for when the phone isn’t in File Transfer / Android Auto mode. So I enabled file transfer (even though the real solution is to fix the udev file, as described below). And that’s when a popup appears asking if the computer should be allowed USB debugging. So yes, and pick “Always allow from this computer”.

Now we’re talking:

$ adb devices
23011xxxxxxxx	device

Yey. Even more yey: Using adb shell, I got

$ id
uid=2000(shell) gid=2000(shell) groups=2000(shell),1004(input),1007(log),1011(adb),1015(sdcard_rw),1028(sdcard_r),1078(ext_data_rw),1079(ext_obb_rw),3001(net_bt_admin),3002(net_bt),3003(inet),3006(net_bw_stats),3009(readproc),3011(uhid) context=u:r:shell:s0

The privileges in this mode is much better than as an SSH client (see below). For example, one can list the files in /system/bin and /dev with adb shell and not through ssh (unless the ssh session gets root). Same goes with using “top”: It shows all processes, not just the same users’, as with ssh (once again, if the ssh session gets root, sky’s the limit).

There are nice executables in /vendor/bin too (even ifconfig and nc)

Getting the udev rule right

To resolve the “permission denied” thing with adb without file system access and fastboot, a udev rule needs to be added.

With MTP, file transfer on and off, I checked which udev rules got in action for the device with

$ udevadm test -a add $(udevadm info -q path -n /dev/bus/usb/002/004)

The 002/004 in the end are the bus and device numbers as found in lsusb.

It turns out that the relevant rule was in /lib/udev/rules.d/69-libmtp.rules:

# Google Inc Nexus/Pixel (MTP+ADB)
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee2", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio", ENV{ID_MTP_DEVICE}="1", ENV{ID_MEDIA_PLAYER}="1"

Note that it sets the group to “audio” and not “plugdev” The ENV{} assignments prevent the call to mtp-probe in this same udev file.

The truth is that I don’t completely understand why that works at all.

Anyhow, I ended up adding the following as /etc/udev/rules.d/20-google-pixel.rules:

# Google Inc Pixel 6 Pro, support for no-file transfer ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee7", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for USB tethering + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4eec", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for MIDI + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee9", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for fastboot mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee0", MODE="660", GROUP="plugdev"

and don’t forget

$ sudo udevadm control --reload

I assigned the group “plugdev” or else fastboot required root (on the host computer) to detect the device. Also, I didn’t add any of the ENV{ID_MTP_DEVICE}=”1″ commands, because doing so prevents running mtp-probe. And oddly enough, if mtp-probe doesn’t run, I get a popup saying “Unable to mount mtp device”.

As a side note, mtp-probe runs on virtually every USB device. This is quite ugly, actually.

Backing up some of the phone’s data

You can’t just use tar to backup the root directory. Not sure why, but nothing happened when I tried. Maybe an SeLinux thing? So it boils down to something like this:

$ time adb shell 'su -c "tar -cz /storage/emulated/0/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/

This backs up the visible user data. For hidden application data, repeat this with data/:

$ time adb shell 'su -c "tar -cz /data/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/

And here’s the crux: There are several duplicate mounts in the filesystem. It seems like a lot under /data/user/0 is a repeated backup. And even more so, /data/media/0 is apparently a duplicate of /storage/emulated/0/ (or the other way around? I’m confused), so maybe there’s no point backing up /storage/emulated/0  if /data is backed up.

For a round-up of backing up things that are probably completely useless:

$ time adb shell 'su -c "tar -cz /mnt/vendor /apex /metadata"' | sudo tar -xvz -C /mnt/tmp/googlepixel/

Have I missed something that should be backed up? I don’t know.

Grabbing system info

Be sure to check out this reference and  this cheat sheet for adb commands.

There’s a utility, dumpsys, that allows getting system information easily from the phone.

So, to tell which application is responsible for a window that just showed up:

$ adb shell dumpsys window windows > windows.txt

Or get some memory info (which application takes how much memory?):

$ adb shell dumpsys meminfo > mem.txt

What caused apps to terminate? This gives a log of last exits for each package.

$ adb shell dumpsys activity exit-info > exit.txt

Or everything at once (takes about 30 seconds to run, and emits a lot of error messages):

$ adb shell dumpsys > all.txt

Note that the file is output on the host, not on the phone with these commands.

To get the system log, use logcat:

$ adb shell logcat -d > all-log.txt

The -d flag tells logcat to terminate when it reaches the end of the log. Otherwise it continues running in “tail -f” style.

To get a list of all running processes:

$ adb shell ps -A > ps-all.txt

Look at this page for some additional utilities, in particular the Package Manager (pm) and Activity Manager (am). For a more in-depth understanding of the machinery, look for information on Intents and Activities.

Also try

$ adb shell device_config list

Bonus: device_config also allows to set the listed parameters, and these settings survive reboot (generally speaking).

List all installed packages:

$ adb shell cmd package list packages > packages.txt

Remove all user data for a package (at adb prompt, com.tencent.mm is WeChat)

pm clear com.tencent.mm

Where apps keep their data

For example, Whatsapp:

• /data/data/com.whatsapp/
• /mnt/installer/0/emulated/0/Android/data/com.whatsapp

The sensitive stuff is in the former. The latter may be accessible to anyone.

System startup

Services that are started during boot are listed in /etc/init/. There’s update_engine.rc and update_verifier.rc services there by the way.

Linux kernel

To download the Linux kernel, go

git clone https://android.googlesource.com/kernel/gs

This is a Generic Kernel Image kernel, meaning that it includes the parts that are common to all devices. Device-specific drivers are applied as kernel modules instead.

The Linux kernel that runs on my machine is commit 740f7fbe5f3917e0a5fa0582b98543af4a15a7ba (with no special tag), which identifies itself as v5.10.43. I deduced the commit based upon the “g740f7fbe5f39″ part in the kernel version that appears in the phone’s about part (5.10.43-android12-9-00005-g740f7fbe5f39).

Note however that some drivers don’t come from this kernel tree, but are rather loaded as modules from other sources. google_charge.c, for example.

SSH session with simpleSSHD

This is a no-cost app, which is essentially the Dropbear server.

Note: It’s also possible to connect with “adb shell” through a plain USB cable, and there’s “adb push” and “adb pull” to transfer files. So the advantage of SSH is limited.

Having SimpleSSHD running and started on the phone, I connected with the address provided on the screen

$ ssh -p 2222 user@10.11.12.13

the password appears on the phone’s screen. To use automatic login, go

cat > ~/authorized_keys

and copy-paste the content of ~/.ssh/id_rsa.pub there. Note that the file on the phone is not under a .ssh/ directory, which is probably why the ssh-copy-id utility doesn’t cut. Note however that once this file is found, a password login is not attempted if the host’s public key doesn’t match, so now try to log in from a computer that doesn’t have one of the listed keys.

In principle, the available executables are in /system/bin. The path contains more directories, but this is the only effective one.

Who am I?

$ id
uid=10225(u0_a225) gid=10225(u0_a225) groups=10225(u0_a225),3003(inet),9997(everybody),20225(u0_a225_cache),50225(all_a225) context=u:r:untrusted_app_29:s0:c225,c256,c512,c768

Backing up directly from phone (command run on receiving host):

$ ssh -p 2222 user@10.11.12.13 tar -cvz /storage/emulated/0/

The execution path…?

$ echo $PATH
/sbin:/system/sbin:/system/bin:/system/xbin

Unfortunately, these directories aren’t readable as a regular user, so it’s impossible to do a plain “ls” on them. Just in case I was looking for excuses to root the phone.

“more” also works fine, but there’s no “less”.

“top” works nicely, but shows only the same user’s processes. There’s also “ps”, but it seems to do the same. But hey, I rooted the phone. So

$ su
# id
uid=0(root) gid=0(root) groups=0(root) context=u:r:magisk:s0

And that’s the reason I rooted the phone, actually.

Recording phone calls

Google has cracked down on phone call recording apps, probably because it’s illegal in some places in the world. Where I live it’s legal to record a phone call you’re participating in, so no problem. The only problem is to find an app that does the job.

So BCR (Basic Call Recorder) is a godsend. It works only on a rooted phone, and mine is, and it’s an open-source no-nonsense, plain and simple app that does one thing: It records phone calls. All of them.

I Installed BCR-1.76 as a Magisk module. After setting up the destination directory and trying to make a call, I got a notification saying “Failed to record call” also saying “This device might not support recording calls associated with the ‘com.android.phone’ app. Following this, I changed the output format to WAV/PCM 48000 Hz, and the recording was successful.

It was OGG/Opus @ 16000 Hz before, and that obviously didn’t work. But I don’t want to generate huge files, so I went for M4A/AAC, 64 kbps and 16000 Hz. And that worked as well. Both sides of the conversation are heard loud and clear, and it works well on and off speaker as well as in the car with bluetooth. Something I wouldn’t mention unless I saw how these things don’t work with the apps available at Google Play. But as BCR gets its voice stream from the natural place in the software environment (which is why root is required), and not with some quirky workarounds, it just works. Plus this app won’t start nagging about upgrades and whatnot. Bliss.

As for installing an app downloaded from the web with root privileges: The project has been running for a few years with its source published on Github, and there’s clearly a community and a userbase of relatively savvy people. So if something was wrong with it, I suppose it wouldn’t have gone unnoticed. I can only speculate on why this is released like this: Possibly this project is part of a spyware suite, so letting legit people use the call recorder helps ensuring that it works on all platforms, as complaints arrive if that isn’t the case.

This is my short war story as I made Xilinx’ Impact, part of ISE 14.7, work on a Linux Mint 19 machine with a v4.15 Linux kernel. I should mention that I already use Vivado on the same machine, so the whole JTAG programming thing was already sorted out, including loading firmware into the USB JTAG adapters, whether it’s a platform cable or an on-board interface. All that was already history. It was Impact that refused to play ball.

In short, what needed to be done:

• Make a symbolic link to activate libusb.
• Make sure that the firmware files are installed, even if they’re not necessary to load the USB devices.
• Make sure Vivado’s hardware manager isn’t running.
• Don’t expect it to always work. It’s JTAG through USB, which is well-known for being cursed since ancient times. Sometimes “Initialize Chain” works right away, sometimes “Cable Auto Connect” is needed to warm it up, and sometimes just restart Impact, unplug and replug everything + recycle power on relevant card. Also apply spider leg powder as necessary with grounded elephant eyeball extract.

And now in painstaking detail.

The USB interface

The initial attempt to talk with the USB JTAG interface failed with a lot of dialog boxes saying something about windrvr6 and this:

PROGRESS_START - Starting Operation.
If you are using the Platform Cable USB, please refer to the USB Cable Installation Guide (UG344) to install the libusb package.
Connecting to cable (Usb Port - USB21).
Checking cable driver.
 Linux release = 4.15.0-20-generic.
WARNING:iMPACT -  Module windrvr6 is not loaded. Please reinstall the cable drivers. See Answer Record 22648.
Cable connection failed.

This is horribly misleading. windrvr6 is a Jungo driver which isn’t supported for anything by ancient kernels. Also, the said Answer Record seems to have been deleted.

Luckily, there’s a libusb interface as well, but it needs to be enabled. More precisely, Impact needs to find a libusb.so file somewhere. Even more precisely, this is some strace output related to its attempts:

openat(AT_FDCWD, "/opt/xilinx/14.7/ISE_DS/ISE//lib/lin64/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/xilinx/14.7/ISE_DS/ISE/lib/lin64/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/xilinx/14.7/ISE_DS/ISE/sysgen/lib/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/xilinx/14.7/ISE_DS/EDK/lib/lin64/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/xilinx/14.7/ISE_DS/common/lib/lin64/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[ ... ]
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libusb.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

It so happens that a libusb module is present among the files installed along with ISE (several times, actually), so it’s enough to just

$ cd /opt/xilinx/14.7/ISE_DS/ISE/lib/lin64/
$ ln -s libusb-1.0.so.0 libusb.so

or alternatively, a symlink to /usr/lib/x86_64-linux-gnu/libusb-1.0.so worked equivalently well on my system.

But then

Trying to initialize the chain I got:

PROGRESS_START - Starting Operation.
Connecting to cable (Usb Port - USB21).
Checking cable driver.
File version of /opt/xilinx/14.7/ISE_DS/ISE/bin/lin64/xusbdfwu.hex = 1030.
 Using libusb.
Please run `source ./setup_pcusb` from the /opt/xilinx/14.7/ISE_DS/ISE//bin/lin64 directory with root privilege to update the firmware. Disconnect and then reconnect the cable from the USB port to complete the driver update.
Cable connection failed.

So yey, it was not going for libusb. But then it refused to go on.

Frankly speaking, I’m not so much into running any script with root privileges, knowing it can mess up things with the working Vivado installation. On my system, there was actually no need, because I had already installed and then removed the cable drivers (as required by ISE).

What happened here was that Impact looked for firmware files somewhere in /etc/hotplug/usb/, assuming that if they didn’t exist, then the USB device must not be loaded with firmware. But it was in my case. And yet, Impact refused on the grounds that the files couldn’t be found.

So I put those files back in place, and Impact was happy again. If you don’t have these files, an ISE Lab Tools installation should do the trick. Note that it also installs udev rules, which is what I wanted to avoid. And also that the installation will fail, because it includes compiling the Jungo driver against the kernel, and there’s some issue with that. But as far as I recall, the kernel thing is attempted last, so the firmware files will be in place. I think.

Or installing them on behalf of Vivado is also fine? Note sure.

Identify failed

Attempting to Cable Auto Connect, I got Identify Failed and a whole range of weird errors. Since I ran Impact from a console, I got stuff like this on the terminal:

ERROR set configuration. strerr=Device or resource busy.
ERROR claiming interface.
ERROR setting interface.
ERROR claiming interface in bulk transfer.
bulk tranfer failed, endpoint=02.
ERROR releasing interface in bulk transfer.
ERROR set configuration. strerr=Device or resource busy.
ERROR claiming interface.
ERROR setting interface.
control tranfer failed.
control tranfer failed.

This time it was a stupid mistake: Vivado’s hardware manager ran at the same time, so the two competed. Device or resource busy or not?

So I just turned off Vivado. And voila. All ran just nicely.

Bonus: Firmware loading confusion

I mentioned that I already had the firmware loading properly set up. So it looked like this in the logs:

Feb 13 11:58:18 kernel: usb 1-5.1.1: new high-speed USB device number 78 using xhci_hcd
Feb 13 11:58:18 kernel: usb 1-5.1.1: New USB device found, idVendor=03fd, idProduct=000d
Feb 13 11:58:18 kernel: usb 1-5.1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Feb 13 11:58:18 systemd-udevd[59619]: Process '/alt-root/sbin/fxload -t fx2 -I /alt-root/etc/hotplug/usb/xusbdfwu.fw/xusb_emb.hex -D ' failed with exit code 255.

immediately followed by:

Feb 13 11:58:25 kernel: usb 1-5.1.1: new high-speed USB device number 80 using xhci_hcd
Feb 13 11:58:25 kernel: usb 1-5.1.1: New USB device found, idVendor=03fd, idProduct=0008
Feb 13 11:58:25 kernel: usb 1-5.1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Feb 13 11:58:25 kernel: usb 1-5.1.1: Product: XILINX
Feb 13 11:58:25 kernel: usb 1-5.1.1: Manufacturer: XILINX

This log contains contradicting messages. On one hand, the device is clearly re-enumerated with a new product ID, indicating that the firmware load went fine. On the other hand, there’s an error message saying fxload failed.

I messed around quite a bit with udev because of this. The problem is that the argument to the -D flag should be the path to the device files of the USB device, and there’s nothing there. In the related udev rule, it says $devnode, which should substitute to exactly that. Why doesn’t it work?

The answer is that it actually does work. For some unclear reason, the relevant udev rule is called a second time, and on that second time $devnode is substituted with nothing. Which is harmless because it fails royally with no device file to poke. Except for that confusing error message.

Using Linux Mint 19, I have a network device that needs DHCP address allocation connected to a USB network dongle. When I plug it in, the device appears, but the DHCP daemon ignored eth2 (the assigned network device name) and didn’t respond to its DHCP discovery packets. But restarting the DHCP server well after plugging in the USB network card solved the issue.

I should mention that I use a vintage DHCP server for this or other reason (not necessarily a good one). There’s a good chance that a systemd-aware DHCP daemon will resynchronize itself following a network hotplug event. It’s evident that avahi-daemon, hostapd, systemd-timesyncd and vmnet-natd trigger some activity as a result of the new network device.

Most notable is systemd-timesyncd, which goes

Nov 11 11:25:59 systemd-timesyncd[1101]: Network configuration changed, trying to establish connection.

twice, once when the new device appears, and a second time when it is configured. See sample kernel log below.

It’s not clear to me how these daemons get their notification on the new network device. I could have dug deeper into this, but ended up with a rather ugly solution. I’m sure this can be done better, but I’ve wasted enough time on this — please comment below if you know how.

Setting up a systemd service

The very systemd way to run a script when a networking device appears is to add a service. Namely, add this file as /etc/systemd/system/eth2-up.service:

[Unit]
Description=Restart dhcp when eth2 is up

[Service]
ExecStart=/bin/sleep 10 ; /bin/systemctl restart my-dhcpd
Type=oneshot

[Install]
WantedBy=sys-subsystem-net-devices-eth2.device

And then activate the service:

# systemctl daemon-reload
# systemctl enable eth2-up

The concept is simple: A on-shot service depends on the relevant device. When it’s up, what’s on ExecStart is run, the DHCP server is restarted, end of story.

I promised ugly, didn’t I: Note the 10 second sleep before kicking off the daemon restart. This is required because the service is launched when the networking device appears, and not when it’s fully configured. So starting the DHCP daemon right away misses the point (or simply put: It doesn’t work).

I guess the DHCP daemon will be restarted one time extra on boot due to this extra service. In that sense, the 10 seconds delay is possible better than restarting it soon after or while it being started by systemd in general.

So with the service activated, this is what the log looks like (the restarting of the DHCP server not included):

Nov 11 11:25:54 kernel: usb 1-12: new high-speed USB device number 125 using xhci_hcd
Nov 11 11:25:54 kernel: usb 1-12: New USB device found, idVendor=0bda, idProduct=8153
Nov 11 11:25:54 kernel: usb 1-12: New USB device strings: Mfr=1, Product=2, SerialNumber=6
Nov 11 11:25:54 kernel: usb 1-12: Product: USB 10/100/1000 LAN
Nov 11 11:25:54 kernel: usb 1-12: Manufacturer: Realtek
Nov 11 11:25:54 kernel: usb 1-12: SerialNumber: 001000001
Nov 11 11:25:55 kernel: usb 1-12: reset high-speed USB device number 125 using xhci_hcd
Nov 11 11:25:55 vmnet-natd[1845]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001002
Nov 11 11:25:55 vmnetBridge[1620]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001002
Nov 11 11:25:55 kernel: r8152 1-12:1.0 eth2: v1.09.9
Nov 11 11:25:55 mtp-probe[59372]: checking bus 1, device 125: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-12"
Nov 11 11:25:55 mtp-probe[59372]: bus: 1, device: 125 was not an MTP device
Nov 11 11:25:55 upowerd[2203]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-12
Nov 11 11:25:55 systemd-networkd[65515]: ppp0: Link is not managed by us
Nov 11 11:25:55 systemd-networkd[65515]: vmnet8: Link is not managed by us
Nov 11 11:25:55 systemd-networkd[65515]: vmnet1: Link is not managed by us
Nov 11 11:25:55 networkd-dispatcher[1140]: WARNING:Unknown index 848 seen, reloading interface list
Nov 11 11:25:55 systemd-networkd[65515]: lo: Link is not managed by us
Nov 11 11:25:55 systemd-networkd[65515]: eth2: IPv6 successfully enabled
Nov 11 11:25:55 systemd[1]: Starting Restart dhcp when eth2 is up...
Nov 11 11:25:55 kernel: IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
Nov 11 11:25:55 vmnet-natd[1845]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001043
Nov 11 11:25:55 vmnetBridge[1620]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001043
Nov 11 11:25:55 vmnetBridge[1620]: Adding interface eth2 index:848
Nov 11 11:25:55 vmnet-natd[1845]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001043
Nov 11 11:25:55 vmnetBridge[1620]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001043
Nov 11 11:25:55 systemd-timesyncd[1101]: Network configuration changed, trying to establish connection.
Nov 11 11:25:55 vmnetBridge[1620]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001003
Nov 11 11:25:55 vmnetBridge[1620]: Removing interface eth2 index:848
Nov 11 11:25:55 vmnet-natd[1845]: RTM_NEWLINK: name:eth2 index:848 flags:0x00001003
Nov 11 11:25:55 upowerd[2203]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-12/1-12:1.0
Nov 11 11:25:55 kernel: IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
Nov 11 11:25:55 systemd-timesyncd[1101]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 11 11:25:55 kernel: userif-3: sent link down event.
Nov 11 11:25:55 kernel: userif-3: sent link up event.
Nov 11 11:25:57 vmnetBridge[1620]: RTM_NEWLINK: name:eth2 index:848 flags:0x00011043
Nov 11 11:25:57 vmnetBridge[1620]: Adding interface eth2 index:848
Nov 11 11:25:57 systemd-networkd[65515]: eth2: Gained carrier
Nov 11 11:25:57 systemd-timesyncd[1101]: Network configuration changed, trying to establish connection.
Nov 11 11:25:57 avahi-daemon[1115]: Joining mDNS multicast group on interface eth2.IPv4 with address 10.20.30.1.
Nov 11 11:25:57 avahi-daemon[1115]: New relevant interface eth2.IPv4 for mDNS.
Nov 11 11:25:57 avahi-daemon[1115]: Registering new address record for 10.20.30.1 on eth2.IPv4.
Nov 11 11:25:57 kernel: r8152 1-12:1.0 eth2: carrier on
Nov 11 11:25:57 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Nov 11 11:25:57 vmnet-natd[1845]: RTM_NEWLINK: name:eth2 index:848 flags:0x00011043
Nov 11 11:25:57 vmnet-natd[1845]: RTM_NEWADDR: index:848, addr:10.20.30.1
Nov 11 11:25:57 systemd-timesyncd[1101]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 11 11:25:58 kernel: userif-3: sent link down event.
Nov 11 11:25:58 kernel: userif-3: sent link up event.
Nov 11 11:25:59 avahi-daemon[1115]: Joining mDNS multicast group on interface eth2.IPv6 with address fe80::2e0:4cff:fe68:71d.
Nov 11 11:25:59 avahi-daemon[1115]: New relevant interface eth2.IPv6 for mDNS.
Nov 11 11:25:59 systemd-networkd[65515]: eth2: Gained IPv6LL
Nov 11 11:25:59 avahi-daemon[1115]: Registering new address record for fe80::2e0:4cff:fe68:71d on eth2.*.
Nov 11 11:25:59 systemd-networkd[65515]: eth2: Configured
Nov 11 11:25:59 systemd-timesyncd[1101]: Network configuration changed, trying to establish connection.
Nov 11 11:25:59 systemd-timesyncd[1101]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).

As emphasized in bold above, there are 4 seconds between the activation of the script and systemd-networkd’s declaration that it’s finished with it.

It would have been much nicer to kick off the script where systemd-timesyncd detects the change for the second time. It would have been much wiser had WantedBy=sys-subsystem-net-devices-eth2.device meant that the target is reached when it’s actually configured. Once again, if someone has an idea, please comment below.

A udev rule instead

The truth is that I started off with a udev rule first, ran into the problem with the DHCP server being restarted too early, and tried to solve it with systemd as shown above, hoping that it would work better. The bottom line is that it’s effectively the same. So here’s the udev rule, which I kept as /etc/udev/rules.d/99-network-dongle.rules:

SUBSYSTEM=="net", ACTION=="add", KERNEL=="eth2", RUN+="/bin/sleep 10 ; /bin/systemctl restart my-dhcpd"

Note that I nail down the device by its name (eth2). It would have been nicer to do it based upon the USB device’s Vendor / Product IDs, however that failed for me. Somehow, it didn’t match for me when using SUBSYSTEM==”usb”. How I ensure the repeatable eth2 name is explained on this post.

Also worth noting that these two commands for getting the udev rule together:

# udevadm test -a add /sys/class/net/eth2
# udevadm info -a /sys/class/net/eth2

Trying to solve a 90 second wait-on-some-start-job on each boot situation, I found that there’s little info on how to tackle a problem like this out there. Most web pages usually go “I did this, hurray it worked!”, but where do you start solving a problem like this if none of the do-this-do-that advice helps?

So this is a random pile of things to try out. Most of the things shown below didn’t solve my own issue, but these are the tools I collected in my toolbox.

I did this on a Debian 8 (Jessie), and the problem was that boot was stuck for a minute and a half on “A start job is running for sys-subsystem-net-devices-eth0.device”. It’s not directly relevant, except that it influences what I tried out, and hence listed below.

I have written a shorter version of this post which focuses on the specific problem. This post is more about the techniques for figuring out what’s going on.

PID 1 at your service

The tricky part of systemd is that much of the activity is done directly by the systemd process, having PID 1. Requests to start and stop services and other units are sent via DBus messages, i.e. over connections to UNIX sockets. To someone who is used to the good-old-systemV Linux, this is voodoo at its worst, but there are simple ways to keep track of this, as shown below.

In particular, don’t strace the “systemctl start” process — it just sends the request over DBus. Rather, attach strace to PID 1, also explained below. That’s where the fork to the actual job process takes place, if at all.

And don’t get confused by having /org/freedesktop/ appearing everywhere in the logs. It doesn’t necessarily have anything to do with the desktop (if such exists), and is likewise relevant to a non-graphical system. DBus’ started as a solution for desktop machines, and that’s the only reason “freedesktop” is everywhere.

First thing first

Read the man page, “man systemd.device” in my case. If there’s another computer with different configuration, see what happens there. What does it look like when it works?

journald -x

As mentioned on this page, if something went wrong during boot, check out the log to see why. The -x flag adds valuable info for solving issues of this sort.

For example,

# journald -x

[ ... ]

May 20 11:41:20 diskless systemd[1]: Job sys-subsystem-net-devices-eth0.device/start timed out.
May 20 11:41:20 diskless systemd[1]: Timed out waiting for device sys-subsystem-net-devices-eth0.device.
-- Subject: Unit sys-subsystem-net-devices-eth0.device has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sys-subsystem-net-devices-eth0.device has failed.
--
-- The result is timeout.
May 20 11:41:20 diskless systemd[1]: Dependency failed for ifup for eth0.
-- Subject: Unit ifup@eth0.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ifup@eth0.service has failed.
--
-- The result is dependency.

Now, how to read it: It’s clear that sys-subsystem-net-devices-eth0.device is the unit that didn’t manage to kick off. But the more important clue is that ifup@eth0.service failed, because it depends on the former.

It’s important, because it explains why an attempt to launch sys-subsystem-net-devices-eth0.device was done in the first place. A lot of “there I fixed it” pages on the web disable the latter service, and get rid of the problem, not necessarily understanding how and why.

So here’s why: On some relatively early systemd versions, *.device units simply won’t launch. It’s worked around by making sure that no other unit requests them. But then some software package isn’t aware of this, and requests a .device unit and there’s the deadlock. Or more precisely, waiting 90s for the timeout.

Kicking if off manually

Obviously, the unit is inactive:

# systemctl status sys-subsystem-net-devices-eth0.device
● sys-subsystem-net-devices-eth0.device
   Loaded: loaded
   Active: inactive (dead)

May 20 12:47:07 diskless systemd[1]: Expecting device sys-subsystem-net-devices-eth0.device...
May 20 12:48:37 diskless systemd[1]: Job sys-subsystem-net-devices-eth0.device/start timed out.
May 20 12:48:37 diskless systemd[1]: Timed out waiting for device sys-subsystem-net-devices-eth0.device.

Note that there’s also systemctl show for a detailed printout of the unit’s assignments.

Try to start it manually (this little session took 90 seconds, right?)

# systemctl start sys-subsystem-net-devices-eth0.device
Job for sys-subsystem-net-devices-eth0.device timed out.

The important takeaway is that we can repeat the problem on a running system (as opposed to a booting one). This allows running some tools for looking at what happens.

On the other hand, I’m not all that sure .device units are supposed to be started or stopped with systemctl at all. Or more likely, that requesting a start or stop on device units means waiting for them to reach the desired state by themselves. This goes along with the observation I made with strace (below), showing that systemd does nothing meaningful until it times out. So most likely, it just looked up the state of the device unit, saw it wasn’t started, and then went to sleep, essentially waiting for a udev event to bring the unit to the desired state, and consequently return a success status to the start request.

In fact, when I tried “systemctl stop” on the eth0 device on another machine, on which it the device file was activated automatically, it got stuck exactly the same way as for starting it on Debian 8.

As far as I understand, these should become active and inactive by a systemd-udev event by virtue of udev labeling. They are there to trigger other units that depend on them, not to be controlled explicitly.

But here comes a major red herring: Curiously enough, during the 90 seconds of waiting, “systemctl starts” created a child process, “/bin/systemd-tty-ask-password-agent –watch”. One can easily be misled into thinking that it’s this child process that blocks the completion of the former command.

So first, let’s convince ourselves that it’s not the problem, because running

# systemctl --no-ask-password start sys-subsystem-net-devices-eth0.device

doesn’t create this second process, but is stuck nevertheless.

This systemd-tty-ask-password-agent process listens for system-wide requests for obtaining a password from the user (e.g. when opening a crypto disk), and does that job if necessary. systemctl launches it just in case, regardless of the unit requested for starting. This is the way to make sure passwords are collected, if so needed. This process is usually not visible, because systemctl commands typically don’t last very long. More about it here.

Actually, checking with strace, systemctl was blocking all those 90 seconds on a ppoll(), waiting for some response from the /run/systemd/private UNIX socket. That’s the DBus connection with process number 1, systemd. In other words, systemctl requested the start of the unit over DBus, and then waited for the result for 90 seconds, at which point it got the answer that the attempt timed out.

DBus

DBus is like a small client-server intranet that is local to the computer. It allows processes that are connected to DBus two features:

• A process can act as a client, and request a function call on another process that is connected to the same bus (acting as a server). When the function call finishes, its result value(s) are returned through the bus, so there’s a real function call look and feel. Both the function call and the result may involve multiple elements of data with various types (strings, booleans, arrays, you name it). DBus has adopted object-oriented teminology, so these function calls are called method calls, and the whole OO jargon (properties in particular) applies.
• A process can broadcast events on the bus, which are called signals, so other processes that are subscribed to this signal get a notification.

DBus was originally developed to allow GUI desktop utilities to talk with each other. Later on, this framework was adopted for more fundamental uses, in particular systemd, Pulseaudio, bluetoothd, cupsd etc. So the “original” Dbus is now called the “session bus”, and the latter is the “system bus”. Two completely independent networks, if you like.

For a more comprehensive explanation, see Wikipedia’s page. I show an example of making method calls on DBus using the command line on this post. For a simple example on how Dbus is used, I suggest this.

Listening to DBus

There’s are two utilities, dbus-monitor and “busctl monitor” for dumping DBus messages (an eavesdrop add-on may be required to allow system-wide message monitoring, but this was not the case on my system). I’ll present monitoring of the system Dbus here. For an example of monitoring a session’s messages, there’s another post of mine.

So on the invocation of

# systemctl start sys-subsystem-net-devices-eth0.device

the output of

# dbus-monitor --system

was

signal sender=org.freedesktop.DBus -> dest=:1.8 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.8"
signal sender=:1.0 -> dest=(null destination) serial=127 path=/org/freedesktop/systemd1; interface=org.freedesktop.systemd1.Manager; member=UnitNew
   string "sys-subsystem-net-devices-eth0.device"
   object path "/org/freedesktop/systemd1/unit/sys_2dsubsystem_2dnet_2ddevices_2deth0_2edevice"
signal sender=:1.0 -> dest=(null destination) serial=128 path=/org/freedesktop/systemd1; interface=org.freedesktop.systemd1.Manager; member=JobNew
   uint32 173
   object path "/org/freedesktop/systemd1/job/173"
   string "sys-subsystem-net-devices-eth0.device"
signal sender=:1.0 -> dest=(null destination) serial=129 path=/org/freedesktop/systemd1/job/173; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.freedesktop.systemd1.Job"
   array [
      dict entry(
         string "State"
         variant             string "running"
      )
   ]
   array [
   ]

and when the timeout occurs with a

Job for sys-subsystem-net-devices-eth0.device timed out.

the following output is captured on the DBus:

signal sender=:1.0 -> dest=(null destination) serial=141 path=/org/freedesktop/systemd1; interface=org.freedesktop.systemd1.Manager; member=JobRemoved
   uint32 173
   object path "/org/freedesktop/systemd1/job/173"
   string "sys-subsystem-net-devices-eth0.device"
   string "timeout"
signal sender=:1.0 -> dest=(null destination) serial=142 path=/org/freedesktop/systemd1; interface=org.freedesktop.systemd1.Manager; member=UnitRemoved
   string "sys-subsystem-net-devices-eth0.device"
   object path "/org/freedesktop/systemd1/unit/sys_2dsubsystem_2dnet_2ddevices_2deth0_2edevice"

Clearly, everything was done by the systemd main process, and almost nothing by the process created from console.

The “sender=:1.0″ part means that the sender is the process number 1 (systemd). Try

$ busctl

(that’s short for “busctl list”) to get a mapping between these addresses and processes.

See the number 173 in the object paths all over in the dbus traffic? That’s the job number as listed in

# systemctl list-jobs
JOB UNIT                                  TYPE  STATE
173 sys-subsystem-net-devices-eth0.device start running

1 jobs listed.

Note that these job numbers have absolutely nothing to do with the Linux PIDs.

Using strace

strace is often very useful for resolving OS problems. It’s however important to realize that the old-fashioned way of stracing the process created on command line will probably not yield much information, because this process only sends a request over DBus.

Instead, strace the process that does the actual work: PID 1, the Mother Of All Processes, the almighty systemd itself. I have to admit that I was first intimidated by the idea to attach strace to this process, but it turns out that it’s usually quite calm, and spits out relatively little unrelated mumbo-jumbo.

Bonus: It’s always the same command:

# strace -p 1 -s 128 -ff -o systemd-trace

This makes a file for each process systemd may fork into. If things went wrong because some process didn’t execute properly, this is how we catch it.

For example, when running the said “systemctl start sys-subsystem-net-devices-eth0.device” command, this was the output:

accept4(12, 0, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 13
getsockopt(13, SOL_SOCKET, SO_PEERCRED, {pid=901, uid=0, gid=0}, [12]) = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 18
read(18, "\270\305\231\206+&\262MI\313[\337y}\314V", 16) = 16
close(18)                               = 0
fcntl(13, F_GETFL)                      = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(13, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
fstat(13, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
setsockopt(13, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
setsockopt(13, SOL_SOCKET, 0x22 /* SO_??? */, [0], 4) = 0
getsockopt(13, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
setsockopt(13, SOL_SOCKET, 0x21 /* SO_??? */, [8388608], 4) = 0
getsockopt(13, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
setsockopt(13, SOL_SOCKET, 0x20 /* SO_??? */, [8388608], 4) = 0
getsockopt(13, SOL_SOCKET, SO_PEERCRED, {pid=901, uid=0, gid=0}, [12]) = 0
fstat(13, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
getsockopt(13, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
getsockname(13, {sa_family=AF_LOCAL, sun_path="/run/systemd/private"}, [23]) = 0
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"\0AUTH EXTERNAL 30\r\nNEGOTIATE_UNIX_FD\r\nBEGIN\r\n", 256}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 45
epoll_ctl(4, EPOLL_CTL_ADD, 13, {0, {u32=2784593808, u64=94857137325968}}) = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 18
read(18, "\10,\360z\t\363+\355D\2556NLkhL", 16) = 16
close(18)                               = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 18
read(18, "$F\3302\215\326\320\251\261\240\217\232\224\1\346\205", 16) = 16
close(18)                               = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 18
read(18, "|\313E\273R\264 \375\v\245\235\206h\247\30-", 16) = 16
close(18)                               = 0
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 18
read(18, "{\200\255\356\26\341b4V_P\225aHkO", 16) = 16
close(18)                               = 0
epoll_ctl(4, EPOLL_CTL_MOD, 13, {EPOLLIN|EPOLLOUT, {u32=2784593808, u64=94857137325968}}) = 0
timerfd_settime(29, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={2519, 883043000}}, NULL) = 0
epoll_wait(4, {{EPOLLOUT, {u32=2784593808, u64=94857137325968}}}, 33, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 359675827}) = 0
timerfd_settime(29, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={2508, 883043000}}, NULL) = 0
epoll_wait(4, {{EPOLLOUT, {u32=2784593808, u64=94857137325968}}}, 33, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 359719924}) = 0
sendmsg(13, {msg_name(0)=NULL, msg_iov(3)=[{"OK b8c599862b26424d89cb5bdf797dcc56\r\nAGREE_UNIX_FD\r\n", 52}, {NULL, 0}, {NULL, 0}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 52
epoll_ctl(4, EPOLL_CTL_MOD, 13, {EPOLLIN, {u32=2784593808, u64=94857137325968}}) = 0
epoll_wait(4, {{EPOLLIN, {u32=2784593808, u64=94857137325968}}}, 33, -1) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 359793419}) = 0
epoll_wait(4, {{EPOLLIN, {u32=2784593808, u64=94857137325968}}}, 33, -1) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 359846496}) = 0
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"l\1\0\0018\0\0\0\1\0\0\0\240\0\0\0\1\1o\0\31\0\0\0", 24}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"/org/freedesktop/systemd1\0\0\0\0\0\0\0\3\1s\0\t\0\0\0StartUnit\0\0\0\0\0\0\0\2\1s\0 \0\0\0org.freedesktop.systemd1.Manager\0\0\0\0\0\0\0\0\6\1s\0\30\0\0\0org.freedesktop."..., 208}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 208
getuid()                                = 0
sendmsg(13, {msg_name(0)=NULL, msg_iov(2)=[{"l\2\1\1&\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0", 32}, {"!\0\0\0/org/freedesktop/systemd1/job/242\0", 38}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 70
sendmsg(13, {msg_name(0)=NULL, msg_iov(2)=[{"l\4\1\1H\0\0\0\2\0\0\0\206\0\0\0\1\1o\0!\0\0\0/org/freedesktop/systemd1/job/242\0\0\0\0\0\0\0\2\1s\0\37\0\0\0org.freedesktop.DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChange"..., 152}, {"\34\0\0\0org.freedesktop.systemd1.Job\0\0\0\0\34\0\0\0\5\0\0\0State\0\1s\0\0\0\0\7\0\0\0running\0\0\0\0\0", 72}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 224
sendmsg(35, {msg_name(0)=NULL, msg_iov(2)=[{"l\4\1\1H\0\0\0\302\0\0\0\206\0\0\0\1\1o\0!\0\0\0/org/freedesktop/systemd1/job/242\0\0\0\0\0\0\0\2\1s\0\37\0\0\0org.freedesktop.DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChange"..., 152}, {"\34\0\0\0org.freedesktop.systemd1.Job\0\0\0\0\34\0\0\0\5\0\0\0State\0\1s\0\0\0\0\7\0\0\0running\0\0\0\0\0", 72}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 224
epoll_wait(4, {{EPOLLIN, {u32=2784593808, u64=94857137325968}}}, 33, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 360201428}) = 0
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"l\1\0\1*\0\0\0\2\0\0\0\227\0\0\0\1\1o\0\31\0\0\0", 24}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"/org/freedesktop/systemd1\0\0\0\0\0\0\0\3\1s\0\7\0\0\0GetUnit\0\2\1s\0 \0\0\0org.freedesktop.systemd1.Manager\0\0\0\0\0\0\0\0\6\1s\0\30\0\0\0org.freedesktop.systemd1"..., 186}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 186
sendmsg(13, {msg_name(0)=NULL, msg_iov(2)=[{"l\2\1\1S\0\0\0\3\0\0\0\17\0\0\0\5\1u\0\2\0\0\0\10\1g\0\1o\0\0", 32}, {"N\0\0\0/org/freedesktop/systemd1/unit/sys_2dsubsystem_2dnet_2ddevices_2deth0_2edevice\0", 83}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 115
epoll_wait(4, {}, 33, 0)                = 0
clock_gettime(CLOCK_BOOTTIME, {2508, 360292582}) = 0
epoll_wait(4, {{EPOLLIN, {u32=2784593808, u64=94857137325968}}}, 33, -1) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 360319860}) = 0
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"l\1\0\0019\0\0\0\3\0\0\0\300\0\0\0\1\1o\0N\0\0\0", 24}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
recvmsg(13, {msg_name(0)=NULL, msg_iov(1)=[{"/org/freedesktop/systemd1/unit/sys_2dsubsystem_2dnet_2ddevices_2deth0_2edevice\0\0\3\1s\0\3\0\0\0Get\0\0\0\0\0\2\1s\0\37\0\0\0org.freedesktop.DBus.Pro"..., 241}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=901, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 241
lstat("/etc", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/etc/systemd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/etc/systemd/system", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/etc/systemd/system/sys-subsystem-net-devices-eth0.device.d", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/run", {st_mode=S_IFDIR|0755, st_size=620, ...}) = 0
lstat("/run/systemd", {st_mode=S_IFDIR|0755, st_size=400, ...}) = 0
lstat("/run/systemd/system", {st_mode=S_IFDIR|0755, st_size=120, ...}) = 0
lstat("/run/systemd/system/sys-subsystem-net-devices-eth0.device.d", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/run", {st_mode=S_IFDIR|0755, st_size=620, ...}) = 0
lstat("/run/systemd", {st_mode=S_IFDIR|0755, st_size=400, ...}) = 0
lstat("/run/systemd/generator", {st_mode=S_IFDIR|0755, st_size=360, ...}) = 0
lstat("/run/systemd/generator/sys-subsystem-net-devices-eth0.device.d", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib/systemd", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/lib/systemd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/lib/systemd/system", {st_mode=S_IFDIR|0755, st_size=36864, ...}) = 0
lstat("/lib/systemd/system/sys-subsystem-net-devices-eth0.device.d", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/lib/systemd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/lib/systemd/system", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
lstat("/run", {st_mode=S_IFDIR|0755, st_size=620, ...}) = 0
lstat("/run/systemd", {st_mode=S_IFDIR|0755, st_size=400, ...}) = 0
lstat("/run/systemd/generator.late", {st_mode=S_IFDIR|0755, st_size=440, ...}) = 0
lstat("/run/systemd/generator.late/sys-subsystem-net-devices-eth0.device.d", 0x7ffec2dd0540) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/systemd/system/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/run/systemd/system/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/run/systemd/generator/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/systemd/system/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/systemd/system/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/systemd/system/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/run/systemd/generator.late/sys-subsystem-net-devices-eth0.device.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
sendmsg(13, {msg_name(0)=NULL, msg_iov(2)=[{"l\2\1\1\10\0\0\0\4\0\0\0\17\0\0\0\5\1u\0\3\0\0\0\10\1g\0\1v\0\0", 32}, {"\1b\0\0\0\0\0\0", 8}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 40
epoll_wait(4, {{EPOLLIN, {u32=3, u64=3}}}, 33, -1) = 1
clock_gettime(CLOCK_BOOTTIME, {2508, 883466886}) = 0
read(29, "\1\0\0\0\0\0\0\0", 8)         = 8
timerfd_settime(29, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={2509, 383043000}}, NULL) = 0
epoll_wait(4, {{EPOLLIN, {u32=3, u64=3}}}, 33, -1) = 1

So what we have here is the acceptance of the connection, sending and receiving messages like those captured with dbus-monitor above, but nothing meaningful was executed: There was no productive system call done, and systemctl didn’t fork. But we can also see which files (actually, directories) systemd was looking for, and didn’t find: It really wanted to find some sys-subsystem-net-devices-eth0.device.d in one of the famous paths. Not that it matters so much, though.

By contrast and for example, if “systemctl start atd” is launched and atd is not already running, systemd (as process 1) forks into another process and calls execve(“/usr/sbin/atd”) on the forked process (after a whole lot of cgroup stuff, closing files etc.). If the same systemctl command is called with the atd service already running, there is no such fork (not surprisingly, systemd does nothing when attempting to start an already started service).

For the record, the failed lookup of directories ins’t the problem: I had the luxury of trying exactly the same on a machine that doesn’t get stuck on starting sys-subsystem-net-devices-eth0.device, and the strace looked the same. Except that the systemd job was terminated immediately and successfully, rather than getting stuck.

On my own behalf, this was the moment I realized that this unit shouldn’t be started at all on the system it gets stuck on.

Checking udev

If a boot problem is related to a device, maybe something went wrong with the device’s bringup, which in turn prevented the relevant .device unit to become active, and then some other unit waits for it…?

So what is running when eth0 is detected?

# udevadm test /sys/class/net/eth0
calling: test
version 215
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

load module index
Network interface NamePolicy= disabled on kernel commandline, ignoring.
timestamp of '/etc/systemd/network' changed
timestamp of '/lib/systemd/network' changed
Parsed configuration file /lib/systemd/network/99-default.link
Created link configuration context.
timestamp of '/etc/udev/rules.d' changed
read rules file: /lib/udev/rules.d/42-usb-hid-pm.rules
read rules file: /lib/udev/rules.d/50-bluetooth-hci-auto-poweron.rules
read rules file: /lib/udev/rules.d/50-firmware.rules
read rules file: /lib/udev/rules.d/50-udev-default.rules
read rules file: /lib/udev/rules.d/55-dm.rules

[ ... ]

read rules file: /etc/udev/rules.d/90-local-imagedisk.rules
read rules file: /lib/udev/rules.d/95-cd-devices.rules
read rules file: /lib/udev/rules.d/95-udev-late.rules
read rules file: /lib/udev/rules.d/97-hid2hci.rules
read rules file: /lib/udev/rules.d/99-systemd.rules
rules contain 393216 bytes tokens (32768 * 12 bytes), 23074 bytes strings
21081 strings (168928 bytes), 18407 de-duplicated (148529 bytes), 2675 trie nodes used
NAME 'eth0' /etc/udev/rules.d/70-persistent-net.rules:2
IMPORT builtin 'net_id' /lib/udev/rules.d/75-net-description.rules:6
IMPORT builtin 'hwdb' /lib/udev/rules.d/75-net-description.rules:12
IMPORT builtin 'path_id' /lib/udev/rules.d/80-net-setup-link.rules:5
IMPORT builtin 'net_setup_link' /lib/udev/rules.d/80-net-setup-link.rules:11
Config file /lib/systemd/network/99-default.link applies to device eth0
RUN 'net.agent' /lib/udev/rules.d/80-networking.rules:1
RUN '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/$name --prefix=/proc/sys/net/ipv4/neigh/$name --prefix=/proc/sys/net/ipv6/conf/$name --prefix=/proc/sys/net/ipv6/neigh/$name' /lib/udev/rules.d/99-systemd.rules:61
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/net/eth0
ID_BUS=pci
ID_MODEL_FROM_DATABASE=RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (Motherboard)
ID_MODEL_ID=0x8168
ID_NET_DRIVER=r8169
ID_NET_NAME_MAC=enx408d5c4d1b15
ID_NET_NAME_PATH=enp1s0
ID_PATH=pci-0000:01:00.0
ID_PATH_TAG=pci-0000_01_00_0
ID_PCI_CLASS_FROM_DATABASE=Network controller
ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Co., Ltd.
ID_VENDOR_ID=0x10ec
IFINDEX=2
INTERFACE=eth0
SUBSYSTEM=net
SYSTEMD_ALIAS=/sys/subsystem/net/devices/eth0
TAGS=:systemd:
USEC_INITIALIZED=20767
run: 'net.agent'
run: '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/eth0 --prefix=/proc/sys/net/ipv4/neigh/eth0 --prefix=/proc/sys/net/ipv6/conf/eth0 --prefix=/proc/sys/net/ipv6/neigh/eth0'
unload module index
Unloaded link configuration context.

Note that the “systemd” tag is in place, and so is the SYSTEMD_ALIAS assignment. So there’s probably no reason udev-wise why there was no .device unit activated.

hwdb.bin

Update the file /etc/udev/hwdb.bin:

# udevadm hwdb --update

Note that it doesn’t touch /lib/udev/hwdb.bin, and I’m unclear how they interact, if at all (“wild” guess: /etc/udev/hwdb.bin overrules the one in /lib/udev, if it exists).

On newer systems it appears to be “systemd-hwdb update”.

After installing wireshark (and tons of packages it depends on) on a rather fresh and bare-boned Debian 8 (Jessie), I got the “A start job is running for sys-subsystem-net-devices-eth0.device” message for a minute and half on every boot.

It was exceptionally difficult to find the reason, because so many packages were installed along with wireshark.

This is the short version of how this was solved. For the entire battery of stuff I tried out, I’ve written a separate post.

Bad omens

# systemctl status sys-subsystem-net-devices-eth0.device
● sys-subsystem-net-devices-eth0.device
   Loaded: loaded
   Active: inactive (dead)

May 20 12:47:07 diskless systemd[1]: Expecting device sys-subsystem-net-devices-eth0.device...
May 20 12:48:37 diskless systemd[1]: Job sys-subsystem-net-devices-eth0.device/start timed out.
May 20 12:48:37 diskless systemd[1]: Timed out waiting for device sys-subsystem-net-devices-eth0.device.

OK. Not surprising it’s not active. So start manually…?

# systemctl start sys-subsystem-net-devices-eth0.device
Job for sys-subsystem-net-devices-eth0.device timed out.

The second line appeared after a minute and a half, of course.

So I went to another, more recent machine (Mint 19) and went

$ systemctl status sys-subsystem-net-devices-eth0.device
● sys-subsystem-net-devices-eth0.device - Killer E2500 Gigabit Ethernet Controll
   Loaded: loaded
   Active: active (plugged) since Wed 2019-02-20 14:48:54 IST; 2 months 30 days
   Device: /sys/devices/pci0000:00/0000:00:1c.2/0000:04:00.0/net/eth0

And then comparing the outputs of just

$ systemctl

it became evident that *.device units are listed on the Mint 19 machine, but not on Debian 8.

Which led me to the conclusion that sys-subsystem-net-devices-eth0.device isn’t meant to be on Debian 8. That the problem isn’t that it’s not starting when commanded to do so, but that it’s not supposed to be started that way. The problem is that some other unit requests it.

As far as I understand, these .device units should become active and inactive by a systemd-udev event by virtue of udev labeling. They are there to trigger other units that depend on them, not to be controlled explicitly. For some reason they aren’t activated on the Debian 8 machine, despite udev rules being roughly the same as on the Mint 19 machine.

In the lack of proper docs (?), I’m left to guess that requesting a start or stop on device units means waiting for them to reach the desired state by themselves. This goes along with an observation I’ve made with strace, showing that systemd does nothing meaningful until it times out. So most likely, it just looked up the state of the device unit, saw it wasn’t started, and then went to sleep, essentially waiting for a udev event to bring the unit to the desired state, and consequently return a success status to the start request.

In fact, when I tried “systemctl stop” on the eth0 device on Mint 19 (i.e. the machine on which it was already loaded) it got stuck exactly the same way as for starting it on Debian 8. So that command probably meant “wait until eth0 goes away”.

Closing in

The trick is now to find which unit causes the attempt to kick off sys-subsystem-net-devices-eth0.device.

# journald -x

[ ... ]

May 20 11:41:20 diskless systemd[1]: Job sys-subsystem-net-devices-eth0.device/start timed out.
May 20 11:41:20 diskless systemd[1]: Timed out waiting for device sys-subsystem-net-devices-eth0.device.
-- Subject: Unit sys-subsystem-net-devices-eth0.device has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sys-subsystem-net-devices-eth0.device has failed.
--
-- The result is timeout.
May 20 11:41:20 diskless systemd[1]: Dependency failed for ifup for eth0.
-- Subject: Unit ifup@eth0.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ifup@eth0.service has failed.
--
-- The result is dependency.

In English: It’s clear that sys-subsystem-net-devices-eth0.device is the unit that didn’t manage to kick off. But the more important clue is that ifup@eth0.service failed, because it depends on the former. The easier solution lies in the latter.

But frankly, I don’t really understand what happened here. If eth0 was detected by systemd, why wasn’t the relevant device unit activated? Or if it wasn’t, why was ifup@eth0.service kicked off? The relevant unit file is a wildcard service, not naming any specific device name.

Solution

The textbook solution is to find why .device files aren’t generated at all on my Debian 8 system, fix that, and then there won’t be any delay. The correct solution in some cases is to manipulate the udev rules, adding a “TAG+=”systemd”" rule to the device, so the device unit is started automatically by systemd (man systemd.device). In my case this tag was already there, so it’s probably some issue with the service that’s supposed to respond to the udev event. So that’s a dead end.

So go the clumsy way: Remove the unit file that requests the device unit (or maybe I should have masked it by adding a file in /etc?). In this case, it’s /lib/systemd/system/ifup@.service, which said:

[Unit]
Description=ifup for %I
After=local-fs.target network-pre.target networking.service systemd-sysctl.service
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
ConditionPathIsDirectory=/run/network
DefaultDependencies=no

[Service]
ExecStart=/sbin/ifup --allow=hotplug %I
ExecStop=/sbin/ifdown %I
RemainAfterExit=true

and then make sure this had no adverse side effects (none found so far). Actually, removing this file can’t be worse than it was when it took 90 seconds to boot, because this service wasn’t launched anyhow, as its precondition never started.

Having A HD-901T2 DVB-T2 stick on a Linux Mint 18.1 running a 4.4.0-53-generic kernel, Tvheadend started painfully slow. It took more than a minute before the web interface was available. And here’s why:

Mar 11 10:22:35 tv systemd[1]: Starting tvheadend.service...
Mar 11 10:22:35 tv tvheadend[6019]:  * Starting Tvheadend tvheadend
Mar 11 10:22:35 tv tvheadend[6019]:    ...done.
Mar 11 10:22:35 tv systemd[1]: Started tvheadend.service.
Mar 11 10:22:35 tv tvheadend[6030]: main: Log started
Mar 11 10:22:35 tv tvheadend[6030]: config: loaded
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-T - loaded 43 regions with 1106 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-S - loaded 1 regions with 112 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-C - loaded 17 regions with 56 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: ATSC - loaded 2 regions with 14 networks
Mar 11 10:22:40 tv tvheadend[6030]: linuxdvb: adapter added /dev/dvb/adapter0
Mar 11 10:22:40 tv kernel: [ 5333.042417] usb 1-4: DVB: adapter 0 frontend 0 frequency 0 out of range (174000000..862000000)
Mar 11 10:24:23 tv tvheadend[6030]: linuxdvb: unable to open /dev/dvb/adapter0/frontend1

So that’s it: Tvheadend fails with /dev/dvb/adapter0/frontend1 somehow, and that failure takes time (the driver’s access timeouts?) and blocks the setup of other functionalities. The thing is that frontend1 is in fact the front end for the Panasonic MN88473 DVB-T/T2/C demodulator, and is redundant, because Tvheadend’s interface with the DVB stick is through /dev/dvb/adapter0/frontend0 (Realtek RTL2832).

The elegant solution is to make the offending device file inaccessible using udev. Since udev doesn’t rename device files that are created by the kernel for a long time, playing with its permissions is the way out. If Tvheadend is denied access to /dev/dvb/adapter0/frontend1, it won’t mess with it.

I have old post on udev hacking, and one on udev rules for a NIC, by the way.

You may skip the following section, and go directly to “The fix”, if the gory details seem boring.

Analyzing the udev situation

Plugging in the DVB stick into the computer with

# udevadm monitor --udev --property

running makes a lot of output, but this among others:

UDEV  [90.760633] add      /devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1 (dvb)
ACTION=add
DEVNAME=/dev/dvb/adapter0/frontend1
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
DVB_ADAPTER_NUM=0
DVB_DEVICE_NUM=1
DVB_DEVICE_TYPE=frontend
ID_FOR_SEAT=dvb-pci-0000_00_14_0-usb-0_4
ID_PATH=pci-0000:00:14.0-usb-0:4
ID_PATH_TAG=pci-0000_00_14_0-usb-0_4
MAJOR=212
MINOR=4
SEQNUM=2829
SUBSYSTEM=dvb
TAGS=:uaccess:seat:
USEC_INITIALIZED=90759736

And looking at the device itself:

$ udevadm info -a -n /dev/dvb/adapter0/frontend1

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-3/dvb/dvb0.frontend1':
    KERNEL=="dvb0.frontend1"
    SUBSYSTEM=="dvb"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-3':
    KERNELS=="1-3"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="00"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="500mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 2"
    ATTRS{bcdDevice}=="0100"
    ATTRS{bmAttributes}=="80"
    ATTRS{busnum}=="1"
    ATTRS{configuration}=="USB2.0-Bulk&Iso"
    ATTRS{devnum}=="3"
    ATTRS{devpath}=="3"
    ATTRS{idProduct}=="0131"
    ATTRS{idVendor}=="15f4"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="astrometadvbt2"
    ATTRS{maxchild}=="0"
    ATTRS{product}=="dvbt2"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="removable"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="18880"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
    KERNELS=="usb1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{authorized_default}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0404"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="1"
    ATTRS{devpath}=="0"
    ATTRS{idProduct}=="0002"
    ATTRS{idVendor}=="1d6b"
    ATTRS{interface_authorized_default}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Linux 4.4.0-53-generic xhci-hcd"
    ATTRS{maxchild}=="7"
    ATTRS{product}=="xHCI Host Controller"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0000:00:14.0"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="52"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0':
    KERNELS=="0000:00:14.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="xhci_hcd"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x0c0330"
    ATTRS{consistent_dma_mask_bits}=="64"
    ATTRS{d3cold_allowed}=="1"
    ATTRS{device}=="0x22b5"
    ATTRS{dma_mask_bits}=="64"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="115"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{local_cpus}=="f"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="-1"
    ATTRS{subsystem_device}=="0x1000"
    ATTRS{subsystem_vendor}=="0x1458"
    ATTRS{vendor}=="0x8086"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

Only the first two entries are interesting: The others climb up through the USB port, the PCIebus and finally the PCI root port.

The fix

So I set up this file as /etc/udev/rules.d/50-dvb-rename.rules

SUBSYSTEM=="dvb", ATTRS{idVendor}=="15f4", ATTRS{idProduct}=="0131", KERNEL=="dvb*.frontend1", MODE:="0000", GROUP:="root"

And refreshed the udev system:

# udevadm control --reload

Don’t get discouraged by /etc/udev/ being almost empty: The systemd rules have moved to /lib/udev/, and following systemd conventions, local fixes should be in /etc/udev/.

That didn’t work exactly as I expected, but good enough (after replugging the DVB stick):

$ ls -l /dev/dvb/adapter0/
total 0
crw-rw----+ 1 root video 212, 0 Mar 11 10:50 demux0
crw-rw----+ 1 root video 212, 1 Mar 11 10:50 dvr0
crw-rw----+ 1 root video 212, 3 Mar 11 10:50 frontend0
c---rw----+ 1 root root  212, 4 Mar 11 10:50 frontend1
crw-rw----+ 1 root video 212, 2 Mar 11 10:50 net0

WTF? Note that the MODE assignment is with a := which is supposed to prevent overruling. And still, it’s group accessible. Where did that come from? Anyhow, that’s why the group is set to root. That fixed the problem with Tvheadend, which now finishes initializing 12 seconds after systemd announces its launch.

More udev rambling

I went on a bit on this. Nothing really for TLDRers.

Tesring (doesn’t really require root privileges, but the actions won’t take place):

# udevadm test -a add $(udevadm info -q path -n /dev/dvb/adapter0/frontend1)

or stating the path explicitly:

# udevadm info -q path -n /dev/dvb/adapter0/frontend1
/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
# udevadm test -a add /devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
[ ... ]
GROUP 0 /etc/udev/rules.d/50-dvb-rename.rules:4
MODE 0 /etc/udev/rules.d/50-dvb-rename.rules:4
IMPORT builtin 'path_id' /lib/udev/rules.d/71-seat.rules:49
RUN 'uaccess' /lib/udev/rules.d/73-seat-late.rules:15
handling device node '/dev/dvb/adapter0/frontend1', devnum=c212:4, mode=0, uid=0
, gid=0
set permissions /dev/dvb/adapter0/frontend1, 020000, uid=0, gid=0
preserve already existing symlink '/dev/char/212:4' to '../dvb/adapter0/frontend
1'
created db file '/run/udev/data/c212:4' for '/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1'
[ ... ]

And here comes the big surprise: After the test, the frontend1′s permission was actually 0000 as required! So who’s fiddling with the permission after udev?

$ ls -l /dev/dvb/adapter0/total 0
crw-rw----+ 1 root video 212, 0 Mar 11 10:50 demux0
crw-rw----+ 1 root video 212, 1 Mar 11 10:50 dvr0
crw-rw----+ 1 root video 212, 3 Mar 11 10:50 frontend0
c---------+ 1 root root  212, 4 Mar 11 10:53 frontend1
crw-rw----+ 1 root video 212, 2 Mar 11 10:50 net0

If anyone knows, please comment below. For my current purpose, I could work this around by changing the group.

And if I’m at it: When Tvheadend runs, all device files in /dev/dvb/adapter0/ appear to have been deleted. They return when the Tvheadend is stopped. Does anyone have an idea on how and why?

Unlike what one might have expected, there was no need to update the initramfs. The attributes of frontend1 were the same (as the replug above, not the test) after a reboot:

# mount -o remount,rw /boot/
# update-initramfs -u -v

which didn’t make any difference, as expected.

I use USB flash sticks for backing up my system periodically by creating an image of the filesystem, and raw-writing it directly to e.g. /dev/sdd1. It’s just a matter of time before I wipe my hard disk by selecting the wrong /dev/something. Or just some other USB stick that happened to be plugged in.

To avoid this, I wrote a udev rule for detecting the two specific USB devices that I use for backing up, and change their ownership, so I won’t be root while doing this (this protects the real disks somewhat). Also, rename the device file to something that can’t be confused with anything else. Ehm, it’s 2018, and systemd’s udev machinery don’t agree renaming /dev files. So add a symbolic link instead. It’s as good, as long as I don’t have to write /dev/sdd-something in the command wiping whatever’s there.

Writing udev rules is well-documented all over the web, but I still wrote down my own quick summary, so I have it handy for the next time. Well, actually twice. The first time was in 2011, and then in 2018, both are on this page. So this is two posts in one.

On systemd-enabled system (that’s all of them by 2018?), the system’s udev rules reside in /lib/udev/. The rules in /etc/udev take precedence, and it’s still the correct place to put local rules. Just don’t be surprised that it’s empty.

And I have another post with udev hacking (and one related to NICs). But now, let’s start with the 2018 post. Updated to Linux Mint 19, with systemd all over the place (and I love it, frankly).

And a little tip to myself and others: man udev. Really. Give it a go every now and then.

Obtaining matching parameters

The built-in udev rules in /lib/udev/rules.d set up a lot of environment variables, which are, among others, printed by e.g.

# udevadm info -q all -n /dev/sdd1

So don’t. It’s better to stick to the actual attributes, as given by

# udevadm info -a -n /dev/sdd1

Note that each segment of this output is a separate device, and hence a separate udev event. All matches in the udev rule must relate to one segment only.

Building the rule

The udevadm info command outputs a very important comment, which is worth repeating:

A rule to match can be composed by the attributes of the device and the attributes from one single parent device.

Based upon the info obtained with udevadm info (see just below), the following rule was set up:

KERNEL=="sd?1", SUBSYSTEM=="block", ATTRS{idProduct}=="5591", ATTRS{idVendor}=="0781", ATTRS{serial}=="1234567890", OWNER:="eli", SYMLINK+="backupstick_a"

One may ask why I bothered to add the match against KERNEL==”sd?1″ and SUBSYSTEM==”block”.

Without these, the match works against e.g. /dev/sdd and /dev/sdd1, even though the symbolic link ends up pointing at /dev/sdd1. However the ownership is set for both, clearly indicating that both sdd and sdd1 were processed. Having some trust in udev, I would expect it to behave in a consistent manner on this, but I don’t want to rely on it. Hence the rule is specific on a first partition (I could have gone ATTR{partition}==”1″ as well, but why).

So the rule above takes advantage of the comment in bold above: Matching from a device and its parent. And it works.

Testing the rules

Note that “udevadm test” performs the actions for real. It’s not a dry run.

There no need for any reload command for testing the rule as follows:

# udevadm test -a add $(udevadm info -q path /dev/sdd) 2>&1 | less

But in order to get it working, a reload is required:

 # udevadm control --reload

Picking the correct subsystem

The SUBSYSTEM match determines which device file the rule will apply to. For example, a USB fax/modem device appears both under the “usb” subsystem, which relates to the /dev/bus/usb/XXX/XXX device, and the “tty” subsystem, in which case it goes with the e.g. /dev/ttyACM0 device.

Likewise, when testing, the rule will take effect on one of the device files, and not the other.

The SYMLINK+= command can be useful to tell what device file is related to, but if the symlink is created, odds are we got it right anyhow…

udevadm info of this case

# udevadm info -a -n /dev/sdd1
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd/sdd1':
    KERNEL=="sdd1"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{alignment_offset}=="0"
    ATTR{discard_alignment}=="0"
    ATTR{inflight}=="       0        0"
    ATTR{partition}=="1"
    ATTR{ro}=="0"
    ATTR{size}=="242614240"
    ATTR{start}=="32"
    ATTR{stat}=="     131        0     6184      308        0        0        0        0        0      204      308"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd':
    KERNELS=="sdd"
    SUBSYSTEMS=="block"
    DRIVERS==""
    ATTRS{alignment_offset}=="0"
    ATTRS{capability}=="51"
    ATTRS{discard_alignment}=="0"
    ATTRS{events}=="media_change"
    ATTRS{events_async}==""
    ATTRS{events_poll_msecs}=="-1"
    ATTRS{ext_range}=="256"
    ATTRS{hidden}=="0"
    ATTRS{inflight}=="       0        0"
    ATTRS{range}=="16"
    ATTRS{removable}=="1"
    ATTRS{ro}=="0"
    ATTRS{size}=="242614272"
    ATTRS{stat}=="     145        0     6296      340        0        0        0        0        0      228      340"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0':
    KERNELS=="8:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    ATTRS{blacklist}==""
    ATTRS{device_blocked}=="0"
    ATTRS{device_busy}=="0"
    ATTRS{dh_state}=="detached"
    ATTRS{eh_timeout}=="10"
    ATTRS{evt_capacity_change_reported}=="0"
    ATTRS{evt_inquiry_change_reported}=="0"
    ATTRS{evt_lun_change_reported}=="0"
    ATTRS{evt_media_change}=="0"
    ATTRS{evt_mode_parameter_change_reported}=="0"
    ATTRS{evt_soft_threshold_reached}=="0"
    ATTRS{inquiry}==""
    ATTRS{iocounterbits}=="32"
    ATTRS{iodone_cnt}=="0x2ac"
    ATTRS{ioerr_cnt}=="0x1"
    ATTRS{iorequest_cnt}=="0x2ac"
    ATTRS{max_sectors}=="240"
    ATTRS{model}=="Ultra USB 3.0   "
    ATTRS{queue_depth}=="1"
    ATTRS{queue_type}=="none"
    ATTRS{rev}=="1.00"
    ATTRS{scsi_level}=="7"
    ATTRS{state}=="running"
    ATTRS{timeout}=="30"
    ATTRS{type}=="0"
    ATTRS{vendor}=="SanDisk "

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0':
    KERNELS=="target8:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8':
    KERNELS=="host8"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0':
    KERNELS=="1-2:1.0"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb-storage"
    ATTRS{authorized}=="1"
    ATTRS{bAlternateSetting}==" 0"
    ATTRS{bInterfaceClass}=="08"
    ATTRS{bInterfaceNumber}=="00"
    ATTRS{bInterfaceProtocol}=="50"
    ATTRS{bInterfaceSubClass}=="06"
    ATTRS{bNumEndpoints}=="02"
    ATTRS{supports_autosuspend}=="1"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2':
    KERNELS=="1-2"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="00"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="224mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0100"
    ATTRS{bmAttributes}=="80"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="10"
    ATTRS{devpath}=="2"
    ATTRS{idProduct}=="5591"
    ATTRS{idVendor}=="0781"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="SanDisk"
    ATTRS{maxchild}=="0"
    ATTRS{product}=="Ultra USB 3.0"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="removable"
    ATTRS{serial}=="1234567890"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="1580"
    ATTRS{version}==" 2.10"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
    KERNELS=="usb1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{authorized_default}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0415"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="1"
    ATTRS{devpath}=="0"
    ATTRS{idProduct}=="0002"
    ATTRS{idVendor}=="1d6b"
    ATTRS{interface_authorized_default}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Linux 4.15.0-20-generic xhci-hcd"
    ATTRS{maxchild}=="16"
    ATTRS{product}=="xHCI Host Controller"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0000:00:14.0"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="134"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0':
    KERNELS=="0000:00:14.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="xhci_hcd"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x0c0330"
    ATTRS{consistent_dma_mask_bits}=="64"
    ATTRS{d3cold_allowed}=="1"
    ATTRS{dbc}=="disabled"
    ATTRS{device}=="0xa2af"
    ATTRS{dma_mask_bits}=="64"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="35"
    ATTRS{local_cpulist}=="0-11"
    ATTRS{local_cpus}=="0,00000000,00000fff"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="0"
    ATTRS{revision}=="0x00"
    ATTRS{subsystem_device}=="0x5007"
    ATTRS{subsystem_vendor}=="0x1458"
    ATTRS{vendor}=="0x8086"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

The post from 2011 starts here

Following this excellent guide, I plugged in the USB stick and went

$ udevadm info -a -n /dev/sdd

Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the udev rules key format. A rule to match, can  be composed by the attributes of the device and the attributes from one single parent device.

On another computer a similar command could be

$ udevinfo -q all -n /dev/sda

The output from the first command was:

 looking at device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0/block/sdd':
 KERNEL=="sdd"
 SUBSYSTEM=="block"
 DRIVER==""
 ATTR{range}=="16"
 ATTR{ext_range}=="256"
 ATTR{removable}=="1"
 ATTR{ro}=="0"
 ATTR{size}=="7821312"
 ATTR{alignment_offset}=="0"
 ATTR{discard_alignment}=="0"
 ATTR{capability}=="51"
 ATTR{stat}=="      49      438     3182      274        0        0        0        0        0      127      274"
 ATTR{inflight}=="       0        0"

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0':
 KERNELS=="15:0:0:0"
 SUBSYSTEMS=="scsi"
 DRIVERS=="sd"
 ATTRS{device_blocked}=="0"
 ATTRS{type}=="0"
 ATTRS{scsi_level}=="3"
 ATTRS{vendor}=="SanDisk "
 ATTRS{model}=="Cruzer Blade    "
 ATTRS{rev}=="1.01"
 ATTRS{state}=="running"
 ATTRS{timeout}=="30"
 ATTRS{iocounterbits}=="32"
 ATTRS{iorequest_cnt}=="0x1a8"
 ATTRS{iodone_cnt}=="0x1a8"
 ATTRS{ioerr_cnt}=="0x1"
 ATTRS{modalias}=="scsi:t-0x00"
 ATTRS{evt_media_change}=="0"
 ATTRS{dh_state}=="detached"
 ATTRS{queue_depth}=="1"
 ATTRS{queue_type}=="none"
 ATTRS{max_sectors}=="240"

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0':
 KERNELS=="target15:0:0"
 SUBSYSTEMS=="scsi"
 DRIVERS==""

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15':
 KERNELS=="host15"
 SUBSYSTEMS=="scsi"
 DRIVERS==""

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0':
 KERNELS=="2-1:1.0"
 SUBSYSTEMS=="usb"
 DRIVERS=="usb-storage"
 ATTRS{bInterfaceNumber}=="00"
 ATTRS{bAlternateSetting}==" 0"
 ATTRS{bNumEndpoints}=="02"
 ATTRS{bInterfaceClass}=="08"
 ATTRS{bInterfaceSubClass}=="06"
 ATTRS{bInterfaceProtocol}=="50"
 ATTRS{modalias}=="usb:v0781p5567d0100dc00dsc00dp00ic08isc06ip50"
 ATTRS{supports_autosuspend}=="0"

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1':
 KERNELS=="2-1"
 SUBSYSTEMS=="usb"
 DRIVERS=="usb"
 ATTRS{configuration}==""
 ATTRS{bNumInterfaces}==" 1"
 ATTRS{bConfigurationValue}=="1"
 ATTRS{bmAttributes}=="80"
 ATTRS{bMaxPower}=="200mA"
 ATTRS{urbnum}=="938"
 ATTRS{idVendor}=="0781"
 ATTRS{idProduct}=="5567"
 ATTRS{bcdDevice}=="0100"
 ATTRS{bDeviceClass}=="00"
 ATTRS{bDeviceSubClass}=="00"
 ATTRS{bDeviceProtocol}=="00"
 ATTRS{bNumConfigurations}=="1"
 ATTRS{bMaxPacketSize0}=="64"
 ATTRS{speed}=="480"
 ATTRS{busnum}=="2"
 ATTRS{devnum}=="8"
 ATTRS{devpath}=="1"
 ATTRS{version}==" 2.00"
 ATTRS{maxchild}=="0"
 ATTRS{quirks}=="0x0"
 ATTRS{avoid_reset_quirk}=="0"
 ATTRS{authorized}=="1"
 ATTRS{manufacturer}=="SanDisk"
 ATTRS{product}=="Cruzer Blade"
 ATTRS{serial}=="200610803009F0712206"

 looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2':
 KERNELS=="usb2"
 SUBSYSTEMS=="usb"
 DRIVERS=="usb"
 ATTRS{configuration}==""
 ATTRS{bNumInterfaces}==" 1"
 ATTRS{bConfigurationValue}=="1"
 ATTRS{bmAttributes}=="e0"
 ATTRS{bMaxPower}=="  0mA"
 ATTRS{urbnum}=="4053"
 ATTRS{idVendor}=="1d6b"
 ATTRS{idProduct}=="0002"
 ATTRS{bcdDevice}=="0206"
 ATTRS{bDeviceClass}=="09"
 ATTRS{bDeviceSubClass}=="00"
 ATTRS{bDeviceProtocol}=="00"
 ATTRS{bNumConfigurations}=="1"
 ATTRS{bMaxPacketSize0}=="64"
 ATTRS{speed}=="480"
 ATTRS{busnum}=="2"
 ATTRS{devnum}=="1"
 ATTRS{devpath}=="0"
 ATTRS{version}==" 2.00"
 ATTRS{maxchild}=="8"
 ATTRS{quirks}=="0x0"
 ATTRS{avoid_reset_quirk}=="0"
 ATTRS{authorized}=="1"
 ATTRS{manufacturer}=="Linux 2.6.35.4-OCHO1 ehci_hcd"
 ATTRS{product}=="EHCI Host Controller"
 ATTRS{serial}=="0000:00:1d.7"
 ATTRS{authorized_default}=="1"

 looking at parent device '/devices/pci0000:00/0000:00:1d.7':
 KERNELS=="0000:00:1d.7"
 SUBSYSTEMS=="pci"
 DRIVERS=="ehci_hcd"
 ATTRS{vendor}=="0x8086"
 ATTRS{device}=="0x3b34"
 ATTRS{subsystem_vendor}=="0x1458"
 ATTRS{subsystem_device}=="0x5006"
 ATTRS{class}=="0x0c0320"
 ATTRS{irq}=="23"
 ATTRS{local_cpus}=="00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff"
 ATTRS{local_cpulist}=="0-7"
 ATTRS{modalias}=="pci:v00008086d00003B34sv00001458sd00005006bc0Csc03i20"
 ATTRS{numa_node}=="-1"
 ATTRS{dma_mask_bits}=="32"
 ATTRS{consistent_dma_mask_bits}=="32"
 ATTRS{broken_parity_status}=="0"
 ATTRS{msi_bus}==""
 ATTRS{companion}==""

 looking at parent device '/devices/pci0000:00':
 KERNELS=="pci0000:00"
 SUBSYSTEMS==""
 DRIVERS==""

From which I extracted the idVendor, idProduct and most important, serial attributes, marked in red above.

So I generated /etc/udev/rules.d/10-local-usbstick-rules saying:

SUBSYSTEMS=="usb", ATTRS{idVendor}=="0781", ATTRS{idProduct}=="5567", ATTRS{serial}=="200610803009F0712206", NAME="usbstick_trials", OWNER="eli"

Note that I set myself as the owner of the device file. This means that the data on the USB stick is not so safe, because I can accidentally write data directly to the device file. The upside is that root privileges are not necessary, so the chances for messing up a real hard disk are significantly diminished.

That’s it. /dev/sdd doesn’t appear now, and neither do partition device files, so it’s a bit more difficult to mount the disk (but there’s a trick). On the other hand, raw writing to it just became much safer.

Plug out, plug in again and we have:

$ ls -l /dev/usbstick_trials
brw-rw----. 1 eli disk 8, 48 2011-06-16 17:54 /dev/usbstick_trials

On the other computer (Centos 5.5) the rule was /etc/udev/rules.d/90-local-imagedisk.rules as follows:

KERNEL=="sda1", ENV{ID_SERIAL}=="SATA_WDC_WD5000AAKX-_WD-WCAYUFH69712", NAME="playdisk", OWNER="eli", MODE="666", OPTIONS="last_rule"

The reason for placing the rule late is to allow 50-udev.rules to set up the ID_SERIAL environment variable, which is created from the disk’s serial number. So no other disk gets messed up this way accidentally

Alternative: Monitor events

# udevadm monitor --udev --property

This prints out the udev events as they happen. The relevant info is there too. Useful when you’re not sure which device belongs to what, but a plug-in event makes it very clear. Unfortunately, it doesn’t print out any actions udev takes as it applies the rules.