Keeping a Gentoo house neat and tidy

11 Oct 2020 at 16:27

admin

Gentoo

No Comments

Semi-automating emerge @world update

tl;dr Use this interactive script on a weekly basis to keep your Gentoo installation up to date and avoid intractable library version conflicts.

Gentoo has a fantastic source-based package management solution, however there are a number of housekeeping details involved in keeping an installation tidy, to avoid increasing issues with conflicts when upgrading large volumes of packages at once. The so-called @world update (upgrading all installed packages at once) is one such case, and it ought to be done regularly in order to avoid having too large a list of packages with updates, which itself increases the risk of conflicts due to differing version requirements of each package requiring upgrade.

The set of steps that need to be run for a clean, housekept @world update are as follows:

  1. Update the local repository of package versions.
  2. Update all packages with updates in the @world set.
  3. Resolve conflicts that prevent packages updating, repeat steps 2 & 3 until all conflicts are resolved.
  4. Uninstall (unmerge in Gentoo parlance) packages that are no longer referenced as dependencies for any package in @world.
  5. Double-check whether a package has been broken due to one or more dependencies having been updated.
  6. Recompile any packages that use a library that has been updated (step 2 already does this to an extent; it is unclear to me if this step is still required).

There seem to be various reasons why Gentoo doesn’t have a combined tool that performs all of the above steps, not least among which is that the approach and individual tools for each step has evolved and been replaced over time. My take is that Gentoo is not for the Linux neophyte and those that use the distro would prefer to have the flexibility of keeping these tools separate and to properly understand what each of them does.

I tend to do an @world update once a week, so I have found it useful to have a script that runs all the various commands in order, logging all the output for reference. I’ve found it always necessary to review this output, both because many packages output important messages that need to be followed up, and because it can contain useful clues when one does run into blocking conflicts that needs to be unpicked. The script I’ve developed is also a dumping ground for the various tips and tricks I have picked up from across the web for resolving the various forms of conflict that can occur.

The script can be found here at my Github repository; it is well-commented to explain the purpose of each of the command switches, and offers break-out pauses after each command is run just in case there’s the need to review what’s been done so far or indeed stop processing if something significant isn’t right.

Installing the EQ10Q equalizer on Gentoo

07 Jul 2020 at 12:00

admin

Gentoo, Linux

No Comments

EQ10Q appears to be the absolute best quality equalizer available for Linux. I have tried that that comes with pulse-effects, but found that it introduced occasional audio glitches; no good for audiophile use.

EQ10Q is an LV2 plugin, so I start from the assumption that you already have JACK installed (doing so is trivial so I will not include instructions here). The following set of steps are what I used to get EQ10Q working:

  1. Emerge dev-cpp/gtkmm, since eq10q depends upon it for it’s UI.
  2. Download and install eq10q (do not use the ebuild available in the darkelf repo).
  3. Attach the audio-overlay repo (layman -a audio-overlay).
  4. Emerge media-sound/carla (requires accepting the live ebuild in package.use, plus enable the gtk2 flag because the eq10q UI uses GTK2).
  5. Run carla and add /usr/local/lib/lv2 to where carla looks for LV2 plugins.

Sync music playlists to Android

27 Apr 2020 at 21:24

admin

Android, Linux

No Comments

tl;dr

MTP sucks hard, and none of the free linux music players seem to have a reliable method for syncing a library subset (such as a playlist) to another device. rsync plus SSH plus ADB (and a little light shell scripting) to the rescue!

the problem

I have music in my library that is not and will never be available on streaming services (well, not until it is cost-effective to stream Plex from the home that is ;o). Consequently I need to have that music copied onto my phone. The set of files is a subset of my entire library and is around 25GB. Wireless syncing is all well and good, but I don’t need the sync to be wireless and wired would be way faster.

the solution

write a shell script that takes the path of your music library and a playlist file as input, then rsync’s the files listed in the playlist over SSH to a localhost port. How in hell is that useful? Well, rather conveniently, ADB allows you to map a local port to one on your connected phone:

adb forward tcp:2222 tcp:2222

So as long as you have an SSH server running on your phone on that port (I use SSHelper) then the files will end up in SDCard/Music on the phone (this path is hard-coded in the script but obvious so can be changed). I get roughly 50MB/s transfer rate, which seems none too shabby; the whole 25GB playlist in around 11 minutes.

Scanning on Gentoo using a Lexmark MX310dn

19 Jan 2020 at 22:05

admin

Gentoo

No Comments

SANE, the de facto linux scanning solution has only one open source driver (backend, in SANE parlance) that supports Lexmark scanner, and only a small number of USB models at that. Lexmark have released a closed-source SANE backend, available at their support site if you search for downloads related to the product you have. Unfortunately they do not offer a portage ebuild, only RPM or DEB packages so it takes a little more effort to get it installed and, at least in my case, needed a few further tweaks to get it operating.

Installing the Lexmark SANE backend

  1. Visit support.lexmark.com and search for your scanner.
  2. On the resulting page, in the Downloads tab, select the most recent Debian release.
  3. Download the offered .deb file (and any firmware update available).
  4. Put the .deb in a folder by itself, then follow ‘Option 2 : manual installation’ of the Arch Linux instructions to install it.

Segfault??

I now found that Xsane would launch and crash, and running the simplest way in to sane:

scanimage - L

would throw a segmentation fault. Great. In my case the fix was to simply disable USB scanner support (no great loss), as follows:

  1. Edit the Lexmark backend configuration file:

sudo nano /etc/sane.d/lexmark_nscan.conf

  1. Find the following variable and set it as below:

DISCOVER_USB_FLAG=0

  1. I also found it useful to make the following setting, else the scanner showed a second entry in the list for the Lexmark:

SHOW_PSEUDO_NETWORK_FLAG=0

Scanner not supported??

So huzzah, now Xsane will open and offer the Lexmark scanner in the list of those detected. When I tried to scan however, after a pause I was getting a window saying that my scanner model wasn’t supported. This seemed highly unlikely. Trying a scan from the command line:

scanimage -d lexmark_nscan > ~/test.jpg

showed me the problem; the scanner just wasn’t listening:

java.net.ConnectException: Connection refused

In my case, it turned out I had two problems; the printer’s firmware needed an update (I reckon it was around 2017 vintage beforehand), and I needed to enable the HTTP server on the printer (dig your way down into the printer’s settings page in the TCP/IP settings). I also enabled HTTPS support (further down the list in the same menu).

It is a shame that the HTTP server is required, given the increased attack surface (printers are hardly known for their strong security after all!) but on the other hand it did provide the most convenient method to update the firmware.

With these two things done I was off to the races. The method of operation is distinctly bizarre (you’ll see), but if you follow the steps it works fine. I also found that if you use the auto document feeder, when you click Scan it runs all the pages through at once but only records the first sheet in a multi-document project. Subsequent clicks of the Scan button however would bring in each successive page.

Done!

Mounting NFS shares when using dhcpcd as a network manager under Gentoo

19 Jan 2020 at 20:44

admin

Gentoo, Linux

No Comments

tl;dr

If you use NFS shares and/or want your workstation to have a static LAN IP and want seamless wired<->wireless LAN roaming under Gentoo/OpenRC, then dhcpcd is your go-to network manager. But not without a bit of manual tweaking..

Use Case

You have the following requirements:

  • your workstation is a laptop that moves between wired and wireless connections within your LAN
  • you use NFS shares that you would like auto-reconnected when roaming between connections, and auto-connected on boot regardless of the connection available at that time (at the very least, do not hang on boot due to unavailable shares)
  • you require your workstation to have a static LAN IP so as to be able to reliably expose services it hosts outside the LAN (i.e. so as to forward ports from your router to it)

Solution

I don’t know if it’s just me, but network roaming under Gentoo using the default netifrc + ifplugd system is just not reliable. Maybe it’s that ifplugd isn’t oriented towards wireless connections, maybe it’s that my usual wired ethernet port is on a USB-C dock, maybe it’s the NFS shares I use, but I was continually finding myself without a path to my LAN when switching from one form of connection to another or have it lock up trying to remount shares. Having bashed my head against this for some weeks I re-read the Gentoo network management article for the millionth time, once again decided against migrating to Network Manager (and its associated systemd leanings), but for the first time noticed that it mentions in passing that the simplest network management solution is to use dhcpcd in Master mode.

So then, I uninstalled ifplugd and the netifrc services in favour of dhcpcd. The latter is admirably install-and-forget, but will not assign a static address to any interface by default. Whilst it can support static IP assignment, in order to use it one needs to specify the configuration for every local interface else confusing badness will transpire1. The approach I chose instead is to use DHCP and apply IP reservation at the DHCP server to maintain a predictable address. Explanation of how to do that is beyond the scope of this article (I don’t know what DHCP server you’re using).

Okay, so now we have a system that will pick up an IP address as it roams from wired to wireless connections and back. Mount-at-boot NFS mounts will not handle this cleanly at all though, so the second part of the magic is to do away with static mounts in /etc/fstab entirely, in favour of using autofs to mount them on demand. Installation and configuration is documented admirably at the Gentoo site, so I will not duplicate that here.

That’s all there is to it! It seems trivial in retrospect, but I put up with all manner of half-working solutions on the way here, so I thought it worth documenting anyway.

 

 

  1. If you have multiple local ethernet ports it’ll assign the same address to all of them regardless of whether they’re even up, resulting in bizarre routing problems. The worst of these issues is that down interfaces receive a metric of 0, that is to say the most preferred, meaning that LAN routing fails because it tries to use interfaces that aren’t up!↩︎

Upgrading the Gentoo kernel when running ZFS on an NVME SSD

01 Jul 2019 at 22:52

admin

Gentoo, Linux

No Comments

Blending together the Gentoo Kernel Upgrade Guide with the excellent Gentoo/ZFS/NVME SSD installation articles by Guy Robot I synthesized the following set of steps, which I am well pleased to say went without a hitch first time :o) Please note this is an upgrade article only and assumes you already have a working system that boots ZFS. If you’re starting from scratch please follow Guy Robot’s articles linked above.
If you’re already there and genuinely only want to upgrade your kernel then please read on..

  1. Obviously enough, start with updating the relevant packages (sys-kernel/linux-firmware, sys-kernel/gentoo-sources, sys-fs/zfs, sys-fs/zfs-kmod; note that since v0.8 ZFS no longer relies upon sys-kernel/spl so this can be unmerged). In my case I just went for a complete emerge update @world:

    emaint -a sync
    emerge –update –deep –changed-use –keep-going @world

  2. Ensure that the file /etc/portage/savedconfig/sys-kernel/linux-firmware points to (or is) a file containing the list of only the firmware blobs to compile into the kernel (the remainder should be commented out or deleted). If you are not doing so already, I recommend setting the savedconfig USE variable for the sys-kernel/linux-firmware package so that future emerges retain your config; your set of required firmware is not likely to change from one kernel build to the next.
  3. The /usr/src/linux symlink points to the kernel source package directory. This needs to be updated to point to the location we wish to use for this build. Emerging the gentoo-sources package with USE=symlink would do this for us automatically, but there are other packages out there that need to know where to find our current source, so it’s best that we manage this manually. The Gentoo way is to use the eselect command:

    eselect kernel list
    eselect kernel set <the index number of the new source dir>

  4. We’ll now do some work in the kernel dir. Change into it explicitly to ensure picking the new location of the symlink:

    cd /usr/src/linux

  5. Copy the currently-running kernel’s config to the new kernel’s source dir to use it as a basis:

    cp /boot/kernels/<currently-running kernel dir>/.config /usr/src/linux/.config

  6. We already have the NVME support enabled from the original install (see Guy Robot’s article for details), so next we need to update the config to match the new kernel version (i.e. bring in options that are new since the last time we built it). In the past I have used make olddefconfig to do this, which supposedly keeps your existing settings and selects sane defaults for any new settings but I got burned badly by it once, so now I use:

    make olddefconfig

    which instead prompts you to select values for any new settings that exist in the new kernel version and not in your old one; with a bit of searching it’s possible to work out what the right option should be for your hardware.

  7. I find it wise to then run the graphical kernel config tool – maybe in browsing through it you’ll find options you now want to enable, or perhaps disable:

    make menuconfig

  8. Right, we’re ready to build the kernel. The time this takes is very variable. It seems that small changes in config mean small build times; I have had it take between a couple of minutes to around 1 hour on my Intel i7-7600U. Set the -j parameter to the number of processor cores you have plus one:

    make -j5

  9. I’m honestly not certain that it’s necessary to rebuild ZFS against the new kernel sources, but since we are about to rebuild the ZFS kernel module it seems prudent:

    emerge sys-fs/zfs

  10. Rebuild the kernel modules (zfs-kmod and any others you’re using) against the new sources:

    emerge @module-rebuild

  11. Install the kernel and modules (they aren’t actually used until we reconfigure grub to know about them, so not to worry):

    make modules_install
    make install

  12. The modules get installed to /lib/modules/<kernel version>, and the kernel and it’s ancilliary files will be installed to /boot. For neatness and the ability to roll back in case of problems we will keep the existing kernel alongside the new, so create subdirs in the form /boot/kernels/<kernel version dir> and move/rename vmlinuz, config and System.map from /boot to there.
  13. Now we need to build an init ramdisk that’ll import our ZFS pools at boot time. First step here is to use Fearedbliss’ tool to create one that is Gentoo/ZFS-friendly (N.B. I’m assuming you’ve already added the necessary Portage overlay to your repos.conf, as explained in Guy Robot’s article)

    emerge bliss-initramfs

    Since v8.1.0, this tool is configured via command-line options plus a settings file at /etc/bliss-initramfs/settings.json. If you are using the systemd init then you can go right ahead, but if you’re using OpenRC then edit the settings to change udevProvider to /sbin/udevd. The command line is as follows in either case:

    bliss-initramfs -k <new kernel version>

    When it has finished, copy the resulting initrd-<kernel version> file from /usr/src/linux to /boot/kernels/<kernel version dir>/initrd

  14. We now need to edit the configuration within the init ramdisk to have it mount our zpools explicitly. Folllow the instructions in Guy Robot’s article starting from “We now need to modify the initramfs.” and up until “Now you have a custom initramfs…” :o)
  15. Next we need to let grub know about our new kernel so we can select it at boot. Edit /boot/grub/grub.cfg to set the timeout to 3 (temporarily until we’re happy with the new kernel) and add a menu item like the below, updating the kernel version number appropriately:

    menuentry “Gentoo – 4.19.52-gentoo-FC.01” {
    linux /@/kernels/4.19.52-gentoo-FC.01/vmlinuz root=rpool/ROOT/gentoo resume=/dev/nvme0n1p3 by=id elevator=noop quiet logo.nologo refresh
    initrd /@/kernels/4.19.52-gentoo-FC.01/initrd
    }

  16. Take a deep breath and test your work:

    reboot

  17. The ‘refresh’ option at the end of the grub menu entry should be used just once – it forces ZFS to purge its cache. We want to have this happen in between one version of the module and another, but not at every reboot, so go back in and edit grub.cfg to remove this option.
  18. Once you are happy with your new kernel, you will want to ensure it’s source code sticks around because some other packages depend on it when building. To prevent it being unmerged when new versions are released, preserve the gentoo-sources package version by adding it to @world:

    emerge –noreplace sys-kernel/gentoo-sources:<version>

  19. You can then do the opposite for the ‘outgoing’ kernel:

    emerge –deselect =sys-kernel/gentoo-sources-<version>

  20. Finally, you’ll need to edit grub.cfg one more time to remove the menu entry for the old kernel version and set timeout back to 0 so that you don’t see the menu on booting. You’re done! Enjoy new kernel goodness :)

OpenRC – managing dependency when you’ve multiple network interfaces

19 May 2019 at 13:05

admin

Gentoo, Linux

No Comments

tl;dr

If you have multiple network interfaces and cannot guarantee that all will always be up, but have other services that depend upon network availability, set rc_depend_strict=”NO” in /etc/rc.conf.

I habitually use a wireless network at home and tether my cellphone whilst out and about. I was finding that the VPN service refused to start when I switch network, complaining that the non-started network was not available. By default, Gentoo openvpn’s rc init script has a depend on ‘net’, which I had assumed was interpreted as ‘any network interface’, but in fact by default in OpenRC this identifier is interpreted as all network interfaces. To change this behaviour edit /etc/rc.conf and set rc_depend_strict=”NO”. Simple as that – net will now be interpreted as any interface instead of all interfaces.

Using ffmpeg to downmix Dolby DTS or 5.1 audio to stereo

25 Apr 2019 at 14:36

admin

Linux

No Comments

So you have a surround sound audio file (DTS, 5.1, quadrophonic, whatever) that you want to play back on your old-skool stereo hi-fi system. It’ll play, but you only get to hear front-left & front-right of the many channels. What’s needed is a way to mix the content of those other channels into FL & FR at an appropriate volume. ffmpeg, our favourite audio swiss-army knife, has the capability built in to downmix to a chosen number of output channel with the -ac option, as follows:

ffmpeg -i input_filename -ac 2 output_filename.wav

I could not however find any documentation on the panning and gain levels this option uses for the surround channels, and to my ear the rear channels are mixed too low (possibly because in my case the source was an old Quadrophonic record release with four equally-utilized channels). Luckily, ffmpeg also exposes the ability to finely tune downmixing, using the ‘pan’ filter. From the excellent and exhaustive documentation (here), plus gain values from someone who trod this path before using mplayer, I synthesised this command line:

ffmpeg -i input_filename -af "pan=stereo|FL<FL+0.5*FC+BL+0.6*SL|FR<FR+0.5*FC+BR+0.6*SR" -acodec pcm_s32le output_filename.wav

Notes:

  1. For quadrophonic source material the ‘+0.5*FC’ and ‘+0.6*SL’ (and right channel equivalents) are superfluous but are included for compatibility with more modern x.1 surround source material.
  2. The output file uses the wav suffix and I am specifying 32 bit samples (ffmpeg passes through sample rate as-is from the source to destination file). Modern multi-channel content is usually ‘hi-rez’, that is to say higher resolution than the usual 16bit/44kHz of CD quality audio and we want to ensure the mixdown does not reduce the quality of the source material; these conversion choices ensure that the quality of the source material is retained . Once you’ve used this approach to downmix to a .wav file you can then use your converter of choice to turn the .wav into a more manageable format such as FLAC or MP3.
  3. If your source material is a DVD-A disc or .iso, the audio files can be found in the AUDIO_TS subfolder. Filename suffixes vary, but you’ll be looking for the huge files in that folder.

References

https://eknet.org/main/linux/downmixdts.html

Configure Firefox to respect local host file domain name entries

08 Apr 2019 at 12:19

admin

Firefox

No Comments

The scenario

You’ve added ‘mynasbox  192.168.1.50’ to your hosts file, you want to be able to type mynasbox in the Firefox URL bar and get to its web-based config UI, but instead you’re seeing a ‘There seems to be a problem with this URL’ error message.

tl;dr

In Firefox about:config, set browser.fixup.dns_first_for_single_words to true to be able to use single-word domain name mappings in your hosts file at the same time as retaining the ability to type searches in the Firefox URL bar.

Update 2020-06-12

The fix described here still works but now has one downside; single-word searches are treated as domain names and do not route (unless of course they’re in your local hosts file).

The Full Story

This topic usually goes off the rails when discussed on ‘net forums because Firefox does not in and of itself directly read your hosts file – it relies upon the operating system to resolve domain names for it. In practice though, things are far from that simple; Firefox has had quite a few features added over the years to assist users in getting to web sites even if they don’t type in a proper domain name, which can mean that what you typed gets invisibly altered before any DNS query takes place, confounding the more technical among us.

There are two features of Firefox config that are at play here – keyword and fixup. Both are accessed by browsing to about:config and accepting the dire warning, then typing the feature name in the search box. The keyword feature seems rather unpopular these days (go search if you’re interested in what it’s for), and though you can achieve the desired result by setting this to false, if you do so you also lose the ability to type searches in the Firefox URL bar. Consequently it’s best to leave this at its default value of true (and indeed this must be the case for the fix below to work correctly).

fixup is the feature where Firefox will try to turn what you type into a parseable URL. It is this that causes the eventual DNS lookup not to match your hosts file entries. Feel free to play with the set of browser.fixup options available, however I find that setting browser.fixup.dns_first_for_single_words to true is the simplest general fix for the problem statement, on the basis that all my local servers have one-word names in the hosts file.

References

https://superuser.com/questions/382905/how-to-prevent-firefox-converting-localhost-urls-into-search-queries

https://support.mozilla.org/en-US/questions/1117449

https://support.mozilla.org/en-US/questions/1011327

Install Gentoo linux on ZFS

07 Jan 2019 at 16:31

admin

Gentoo, Linux

No Comments

Gentoo, being a source-based linux distribution, has a very particular installation process. One has to make many decisions at every step along the way, since installation and configuration effectively happen at the same time, and certain things can be very hard to change once the installation has completed. The prime example of this hard-to-change-after-the-fact situation is the choice of file system on which the OS is being installed. Whilst there are a variety of filesystems available for use under linux, in my opinion ZFS remains the best one available, at least until BtrFS matures and becomes stable enough for use in a production environment.

The Gentoo article on the topic of installing on ZFS is very good, but out of date in that it does not cover the extra steps required when installing on an NVMe SSD. My life was saved by a blogger named Guy Robot, whose excellent article I followed to achieve success. Thanks Guy!

← Older Entries