First of all, I wish you a happy 2022! Will this year be the year of the Linux Phone?

Malware on the Pine64 PinePhone

A few months ago, an arbitrary command execution exploit was discovered in the Quectel modem firmware running on the Pine64 PinePhone and PinePhone Pro modems. This exploit was reported to Quectel, got CVE-2021-31698 assigned, and a detailed write-up was published. This exploit allows to execute any command as root on the modem through AT commands, which could be misused by malware running on the PinePhone.

On 5/12/2021, a user shared a binary in the Pine64 channels of a Snake game. As usual, never ever run random binaries on your devices, even if you get a link to the source code. Always build binaries from source as you can never be sure what is inside a binary. However, some people still executed this binary on their PinePhones which installed a systemd job to fully wipe your system and modem. A detailed write-up of the malware situation is explained in a Hackaday blog post.

As a postmarketOS developer and user, I would never be affected by the systemd job since postmarketOS uses OpenRC instead of systemd. Nevertherless, this modem exploit is a serious security vulnerability because it can be used to turn your modem into a brick. Ollieparanoid warned already for outdated firmware in 2017 (!) in a blog post and what can(not) be done about it.

What are our options to avoid this?

  1. Quectel releases a new version of their firmware. However, this has not happened since they confirmed the vulnerability on 13/04/2021.
  2. Patch the vulnerability ourselves with custom firmware. Such firmware exists on Github which is written by Biktorgj!

Option 2 seems the way forward, but how can you deploy this fix to end-users? You cannot just upgrade a package through a package manager, you need to flash this firmware safely on the end-user’s modem.

fwupd to the rescue!

As a postmarketOS developer, I follow its philosophy of leveraging existing technologies and upstreaming changes, instead of reinventing the wheel (https://wiki.postmarketos.org/wiki/About_postmarketOS#Sustainable_and_maintainable). After some investigating, I discovered fwupd which is a specialized daemon for upgrading firmware on hardware. It is commonly used by hardware vendors which support Linux on their hardware to distribute firmware and upgrade it on the end-user hardware. Therefore, fwupd is:

  • Battle-tested by a lot of hardware vendors and users.
  • Integrated with GNOME Software and KDE Discover, but provides also an CLI client.
  • Handles safety features such as checksums, battery level, etc.
  • Supports ModemManager out-of-the-box.
  • Root permissions are only used when really needed such as during flashing.
  • Multiple firmwares can exist for the same hardware through firmware branches. This way, you can easily revert to the original Quectel firmware if necessary.

Installing fwupd and enabling local repositories

fwupd is already packaged for postmarketOS in the Alpine Linux repos, so we can just install it:

pine64-pinephone:~$ apk add fwupd fwupd-plugin-all

After installing fwupd and its plugins on my PinePhone running postmarketOS, I could see which devices are detected:

pine64-pinephone:~$ fwupdmgr get-devices

Pine64 PinePhone (1.2)
│
├─DA4032:
│     Device ID:          0e0c93c8b4bb222157feedbde8f863e23bd1a8f7
│     Current version:    0x3034313430363139
│     Vendor:             EMMC:0x000045
│     GUIDs:              8c6f96d4-f406-53fa-84ca-a367cd368dfc
│                         add792f2-7053-5638-80e7-20987fd16769
│                         5500994c-1b73-52b4-acf4-bb574b6f3029
│     Device Flags:       • Internal device
│                         • Updatable
│   
└─QUECTEL Mobile Broadband Module:
      Device ID:          976c4a39e87f61e6940ea6a8d39c583cfa99615f
      Summary:            Quectel EG25-G modem
      Current version:    0.5.1
      Vendor:             QUALCOMM INCORPORATED (USB:0x2C7C)
      Release Branch:     FOSS-002
      GUIDs:              db379a33-254f-5140-b37e-d36ae7e5c039
                          1a2996cb-f86e-5583-a464-e1b96e1c6ae9
                          587bf468-6859-5522-93a7-6cce552a0aa3
                          22ae45db-f68e-5c55-9c02-4557dca238ec
      Device Flags:       • Updatable

Great! The modem is already seen by fwupd! However, it is not supported by LVFS, the Linux Vendor Firmware Service. LVFS is the default repository of fwupd where it finds and downloads firmware for your hardware. Even though Quectel is on LVFS, it doesn’t provide any firmware packages for the Quectel EG25-G modem in the Pine64 PinePhone and PinePhone Pro. Fortunately, we can enable local repositories as well, such as vendor-directory. In such a local repository, we can drop update packages and fwupd will use them to upgrade the hardware.

I edited the file at /etc/fwupd/remotes.d/vendor-directory.conf to set Enabled to true:

pine64-pinephone:~$ cat /etc/fwupd/remotes.d/vendor-directory.conf
[fwupd Remote]
# this remote provides dynamically generated metadata shipped by the OS vendor and can
# be found in /usr/share/fwupd/remotes.d/vendor/firmware
Enabled=true
Title=postmarketOS (Automatic)
Keyring=none
MetadataURI=file:///usr/share/fwupd/remotes.d/vendor/firmware
ApprovalRequired=false

We have now a local repository enabled, but still no firmware packages in it, let’s change that!

Creating firmware packages

Packaging firmware for fwupd is mostly done by vendors instead of end-users. Most information about packaging firmware for fwupd can be found in the LVFS documentation. The most important part in the documentation is the metadata documentation which describes how you must describe your upgrade package in an XML file. This way, LVFS and fwupd know which version is inside the package, to which hardware it applies to, etc.

Once you have written your metadata, you can package your firmware, which heavily depends on the upgrade protocol. In our case, we make use of the fastboot protocol advertised as com.google.fastboot in the metadata. This protocol assumes that you provide a ZIP archive containing an XML file with the partitions of your fastboot device and its corresponding images:

ZIP FILE
├── appsboot.mbn
├── boot-mdm9607.img
├── NON-HLOS.ubi
├── partition_nand.xml
├── recoveryfs.ubi
├── recovery.img
└── rootfs-mdm9607.ubi

The partitions are described in partition_nand.xml which looks like this:

<?xml version="1.0" encoding="utf-8"?>
<nandboot>
	<magic_numbers>
		<usr_part_magic1>0xAA7D1B9A</usr_part_magic1>
		<usr_part_magic2>0x1F7D48BC</usr_part_magic2>
	</magic_numbers>
	<partition_version length="4">0x4</partition_version>
	<partitions>
		<partition>
			<name length="16" type="string">0:$FASTBOOT_PARTITION_CMD</name>
			<size_kb length="4">1280</size_kb>
			<pad_kb length="4">256</pad_kb>
			<which_flash>0</which_flash>
			<attr>0xFF</attr>
			<attr>0x01</attr>
			<attr>0x00</attr>
			<attr>0xFF</attr>
            		<img_name type="string">$FILE_TO_FLASH</img_name>
            	</partition>
		<! -- More partitions here, removed them for readability -->
	</partitions>
</nandboot>

Each partition has a fastboot partition command specified such as recoveryfs, boot, etc. associated with a file to flash from the ZIP file.

fwupd’s fastboot plugin reads this XML file and flashes each image to its corresponding partition on the modem through the fastboot protocol. However, how do you convince the Quectel EG25-G modem to expose itself as a fastboot device? That’s where ModemManager comes in!

ModemManager inhibiting and fastboot mode

Modemmanager provides a Firmware DBus interface which exposes the firmware information about the modem such as the firmware version and the AT command to enter fastboot mode. Moreover, ModemManager also provides a way to inhibit a modem it is exposing. This way, ModemManager can handover the controls to other software such as fwupd. When fwupd wants to put the modem into fastboot, it asks ModemManager the AT command to send to the modem and also to leave the modem alone by inhibiting it.

The modem reboots into fastboot mode and fwupd can upgrade the modem with the firmware of the ZIP file from above. Once completed, fwupd uninhibit the modem in ModemManager. ModemManager will probe the modem again and expose it over DBus.

Sounds good right? If it would work out-of-the-box, which it doesn’t!

Patching the stack

After weeks of debugging I discovered the following bugs:

  • fwupd:
    • Quectel EG25-G modem only accepts USB packets with a size of 16384 bytes, otherwise the next packet will cause a reboot. This was discovered using WireShark and the Linux kernel’s usbmon feature.
    • Quectel EG25-G modem is too slow to keep up with fwupd, causing random lockups. Enabling verbose logging in fwupd made it slow enough :smile:
    • Quectel EG25-G modem exposes itself as an Android smartphone in fastboot mode instead of using the Quectel USB vendor and device IDs.
    • fwupd does not uninhibit a modem when it comes back after an upgrade. ModemManager sees the modem but doesn’t expose it because of it.
    • fwupd does not remember the firmware branch currently running on the modem.
    • fwupd does not require AC power when upgrading modems. Updating on battery-only is allowed.
  • ModemManager:
    • Quectel modems use a non-standard AT command to get the full firmware version (AT+QGMR) which is not used by the Quectel plugin in ModemManager.
  • eg25-manager:
    • Kills the upgrade because it thinks the modem is lost.
  • GNOME Firmware Updater:
    • Does not support firmware branches
    • Is not adaptive for phone screens

Since these bugs cross multiple FOSS projects, it involves a lot of collaboration between various maintainers when submitting patches, but I upstreamed everything:

  • fwupd:
    • Initial support: MR 4076
    • Stability and safety improvements such as AC power requirement: MR 4086
    • ModemManager not unhibited fix: MR 4112
    • Branch support for modem-manager plugin: MR 4133
  • GNOME Software in Alpine Linux
    • Enable fwupd for all arches on Alpine Linux: MR 28536
  • eg25-manager:
    • Do not reset the modem during upgrade: MR 44
  • GNOME Firmware Updater:
    • Firmware branch support: fixed by the maintainer, thanks Richard Hughes!
    • Make adaptive: MR 66
  • ModemManager:
    • Ignore carrier config version: MR 712
    • Use custom Quectel AT command to get firmware version: MR 713

Most of these patches are being merged upstream or will be soon. Since I upstream all my patches, all distros will benefit from these changes! This upstream-first approach is, in my opinion, the only sustainable approach if we want to make a success of Linux Mobile!

I’m hyped! Can I test this?

I have an open MR at postmarketOS to implement this feature: pmaports MR 2760

However, I cannot make the firmware packages public yet since there’s a discussion going on between Pine64 and Quectel regarding the distribution of the firmware. The problem is that the Hexagon firmware of the ADSP (baseband) is still closed source, and we don’t have any permission yet to distribute this file in our firmware packages yet. However, I made a demo for those who are curious to see how it works!

Demo video on PeerTube

Once the firmware distribution and license discussion is resolved, this feature will land in postmarketOS edge! Stay tuned!