Python pip install -r requirements.txt throws “error: externally-managed-environment”

04 Oct 2025 at 18:18

admin

Uncategorized

No Comments

Background: There are a couple of excellent posts on Stack Overflow that explain why “error: externally-managed-environment” is thrown when trying to execute pip install -r requirements.txt – in modern Python one must always create a virtual environment for each project so as to install it’s dependencies locally and avoid clashing with Linux’ own package manager. In my case however I still kept getting the same error even after having created and activated my project’s own virtual environment. This post then is an aide memoire to myself and to help anyone else that end up in the same situation.

Steps to set up a basic Python environment from scratch on a fresh Ubuntu install

The Ubuntu guidance page tells you to run sudo apt install -y python3-full, which might be simple but also brings with it all the class libraries plus the graphical IDE. If you don’t want all that, the following will provide you a minimal environment and use several hundred MB less disk space :

  1. Update the package lists: sudo apt-get update
  2. Upgrade what’s installed: sudo apt-get upgrade
  3. Install the Python package manager: sudo apt install -y python3-pip
  4. If any packages failed to install, fix missing: sudo apt-get install -y --fix-missing python3-pip
  5. Install virtual environment (venv) support: sudo apt install python3.12-venv

Steps to set up a virtual environment for a given existing Python project

As mentioned in the preamble, the modern way is to create separate virtual environments (venvs) for each project you’re working on, so as to install that project’s dependencies separately from the system Python (and from other projects). To do so for an existing project:

  1. Change to the project’s working directory: cd <path to project>
  2. Create a virtual environment there: python3 -m venv <venv name>. This will be created as a subfolder within the project directory, so keep it short.
  3. Activate the venv: source <venv name>/bin/activate. You’ll notice the command prompt will now be prefixed with (<venv name>)
  4. Install pip in the venv: python -m pip install --upgrade pip. This is the step that was missing in other guides I found.
  5. Confirm that the venv has it’s own Python: which python. It should list the Python interpreter installed within <venv name>/bin and not /usr/bin.
  6. Confirm that the venv has it’s own pip: which pip. It should list the pip executable installed within <venv name>/bin and not /usr/bin.

Note that once working within the venv, sudo is no longer required (perhaps explicitly should not be used)

Finally…

With all the above in place, I could finally run pip install -r requirements.txt and not get the dreaded “externally-managed-environment” message.

Setting permissions on TrueNAS file shares whilst supporting NFS & SMB clients

12 Jan 2025 at 12:49

admin

FreeNAS, Networking, TrueNAS Core

No Comments

In my environment I need to support NFS clients (NFS is faster – good for streaming) and also SMB clients (Mac & Windows). The following notes are the results of my experimentation in using different methods for restricting access to the shares.

NFS shares depend upon user UIDs matching, and permissions only seem to work if POSIX permissions are used. If I apply ACLs to the share – even one named for my user – I lose access unless I include a very open Everyone@ ACL.

I am therefore restricted to using simple (POSIX) permissions, with me (the administrator of the environment) as owner and the share content owner as a group name. I then put the share content owner’s user in that group. Using this approach, there is no need to set a mapall or maproot for NFS shares – just enable the share and away we go. One side effect of this approach though is that I had to update Transmission’s default umask from 018 to 002, else files and folders it creates would be read-only for all users other than transmission itself.

SMB shares are then simply user-based – I do not apply anything in the Filesystem ACLs. As for the Share ACL that is to be experimented with – perhaps it controls whether the share is visible to end users at all?

Achieving flow-state whilst working

21 Dec 2024 at 12:40

admin

Uncategorized

No Comments

As many others have discovered, I find that background music can be an excellent way to get into ‘the zone’ whilst working, where what you’re doing flows effortlessly from your fingertips and everything just clicks. The only issue I’ve run into is that selecting an album becomes a whole thing in itself, and the fact that about 45 minutes later I’ll have to do it all over again!

A few years ago I found an excellent solution; online noise generators. After some research I found what must be the absolute pinnacle of the genre: mynoise.net. The guy behind the site clearly wants everything to be done in the best manner possible. He goes on field trips to make high quality original recordings, and he has combined his immense library of sounds into soundscapes. The special sauce is that one can automate the volume sliders of the set of sounds in a generator, resulting in an infinitely ever-changing background of sound; no more changing the album after 45 minutes! What’s amazing is that, due to the sound quality and the variation, the generators never become intrusive or annoying – I have had a single generator running during an entire working day before now. They just sink into the background, keeping that kitten busy and allowing me to focus and get on with whatever I need to do.

My go-to generator is The Name Of The Rose – you find yourself in the chapel of a 14th century monastery – but there are literally hundreds of other generators to choose from – windscapes, seascapes, cafés, music, marketplaces, sci-fi – the list is practically endless! Add to this the thoughtful site design and you have a rare example of the internet like it used to be – no ads (the site is supported entirely subscription), no distractions, just a web site that’s the result of one person’s passion. Long may it continue to enable flow-state!

 

Setting up custom buttons and scrolling for Kensington trackballs under Linux part 2 (Wayland version)

01 Jul 2024 at 18:04

admin

Gentoo, Linux

1 Comment

In an earlier article I documented how I set up a custom configuration for my trackball’s buttons and scroll direction on Linux when using the X11 windows server. KDE Plasma 6 brings the Wayland window server to my Linux installation, replacing X11, so I’ve had to think again how to implement my custom config.

The general design philosophy of Wayland seems to be that it’s up to the desktop environment (such as KDE Plasma or Gnome) to provide configuration options – Wayland itself is very light. Plasma 6’s System Settings supports left-handed mode (that is, switching the behaviour of the left and right buttons) and rotating the ball to scroll works out of the box, though it is not possible to alter the scroll direction. The deal-breaker though was lack of support for Click Lock AKA Drag Lock (a button to ‘pick up’  items so they can be moved without holding down a trackball button).

First solution using input-remapper

I initially achieved Drag Lock using input-remapper, writing this macro as the output for the selected button :

if_eq($a, 0, key_down(BTN_LEFT).set(a,1),key_up(BTN_LEFT).set(a,0))

It defines a variable ‘a’ to keep state. If a==0 then send the ‘left button down’ event and set a=1. If a==1 then send the ‘left button up’ event and set a=0.

Input-remapper however entirely disregards the behaviour configured in Plasma System Settings, so I needed to additionally configure swapping left and right buttons. and I was unsuccessful in using it to reverse the scroll direction. Additionally, input-remapper depends on systemd to configure it as a background service whereas I still use SysV init, so this solution saw me needing to manually reapply the mapping every time I switched the trackball to the Gentoo machine which would get very old very quickly.

Better solution using evsieve

As I kept googling around for solutions I came across the rather superb evsieve project. On Linux, all input devices send input events via the evdev protocol. evsieve is a pure command-line utility for filtering and altering that stream of events. Its command set is very extensive and very well thought-through, covering all manner of special and edge cases. My solution was as follows:

evsieve --input /dev/input/by-id/usb-047d_Kensington_Slimblade_Trackball-event-mouse grab persist=reopen \
--map yield btn:left btn:right \
--map btn:right btn:left \
--hook btn:middle toggle \
--toggle btn:middle btn:left:0 btn:left:1 \
--output

Explaining each element in turn:

  • the –input line selects the device we wish to monitor for events. ‘grab’ means that evsieve is the exclusive consumer of that device’s events; the desktop environment only sees evsieve’s output. Events that aren’t part of the mapping pass through unchanged, whilst events that match the mapping get altered. ‘persist=reopen’ is a killer feature – it means that the configuration remains in place even if the input device disappears. In my case, where I use KVM switch, this saves me having to get into udev event detection to set up evsieve every time I switch devices.
  • the first –map line turns left button clicks into right button clicks. Since evsieve works such that, by default, the output of each parameter act as input to the following one, ‘yield’ breaks that chain. Without this, the subsequent mapping would just reverse this one.
  • the second –map line turns right button clicks into left button clicks.
  • the –hook line is used to flip the toggle (described in the next parameter) whenever the middle button is clicked (the top-left button on Kensington trackballs sends the middle button event).
  • the –toggle parameter sends either a left button down or left button up event when the middle button is clicked, depending upon the state of the toggle.
  • the –output line is unconfigured, which means the complete stream of output events gets sent from a virtual input device that evsieve sets up automatically.

Since altering the event stream requires root privileges, and because I want this running all the time, I set up a SysV init wrapper and installed it as a service to call the command line above during boot. A systemd equivalent is left as an exercise for the reader :)

Conclusion

This configuration is surprisingly simpler and more reliable than the solution I’d arrived at for X11, even though Wayland offers no features to assist at all. I have given in on the scroll wheel direction since it was not obvious how to achieve my previous preference, but the muscle memory reprogramming only took a couple of days to kick in. I am a happy camper :)

Offsite backup with TrueNAS Core, Duplicacy and an SFTP destination with public key authentication

17 May 2024 at 10:32

admin

FreeNAS, Linux, TrueNAS Core

No Comments

Here’s the process to set up duplicacy to send backup data to a remote SFTP server, with unattended operation supported through using public key authentication. I will assume you already have:

  • an account set up on the destination server
  • generated an ssh keypair with ssh-keygen and NOT specified a passphrase (or add the passphrase in the preferences file for unattended backup, using duplicacy set -key ssh_passphrase -value <passphrase>)
  • the public key has been copied into the ~/.ssh/authorized_keys file on the server
  • the private key file is in ~/.ssh on your source system

If you already have duplicacy set up and backing up to an alternate location, you can retain but disable that config by renaming the .duplicacy subfolder of wherever duplicacy is executed from. Yes, one can support multiple configurations within one preferences file by using the -storage-name init parameter for each setup, but I prefer entirely separating the configs, though I still use -storage-name to remind myself what each one is for.

  1. First of all, change into the root folder of your source data to be backed up, then initialise duplicacy with the following command. Note that -encrypt only encrypts the config file stored server-side. If you wish to fully encrypt the backup, you’ll need to create a second keypair that must use the RSA algorithm, ensure the public key is in PEM format and add -key <path/to/public-key-filename.pem> to the command: 
    duplicacy init -encrypt -storage-name <name-for-your-SFTP-destination> <name-for-your-source-data> sftp://<username>@<server-domain-name-or-IP-address>:<port-number-SSH-is-listening-on>/<subfolder-in-which-to-store-the-backup>
  2. You’ll be prompted for the encryption password.
  3. Store the encryption password in your preferences file so that it can run unattended when called by cron/periodic in future: 
    duplicacy set -key password -value <your-encryption-password>
  4. Store the path to your private authentication key in your preferences file so that duplicacy knows which key to use for authentication: 
    duplicacy set -key ssh_key_file -value <path/to/private-key-file>
  5. Since your .duplicacy/preferences file now stores secrets, make sure it is only read-write to whichever user account will run duplicacy during unattended backups. By default this would be root.
  6. Test your setup by running a dry-run backup. Although this won’t copy any files, if you have a large data set (hundreds of thousands of files) be aware that the first phase of the process, indexing, may well take over an hour alone. As a rough idea, my system with ~400,000 files for ~4TB takes 1.5 hours to index: 
    duplicacy backup -dry-run -stats
  7. If your test run all goes well you can run your initial backup. Be aware that for a large dataset (>1TB) over the average internet connection this will take multiple days! You can speed up the file copy portion of the process by using -threads which will open multiple connections to the SFTP server, the cost being increased memory usage on the client side. With only four threads my system peaked at around 8GB being used by duplicacy’s process, so be careful to ensure you have the memory headroom, else the process will slow to a crawl. 
    duplicacy backup -threads 4 -stats
  8. To configure unattended backup, follow the appropriate portions of this article to set up periodic and the set of scripts the article provides to invoke your backup config. Obviously enough, ignore the parts about duplicacy initialization to set up BackBlaze as destination!
  9. I recommend adding >> backup.log 2>&1 to your scripted duplicacy invocation so as to log its output to a file. That way you can periodically review the log to see what data is being copied and how long each backup is taking, such that you can tweak the number of threads up if you think it could afford to perform copying more quickly, or down if it seems to be browning out on memory.

Compact disc player S/PDIF input for Moode Audio on Raspberry Pi

12 Mar 2024 at 14:46

admin

Audio, Linux

4 Comments

Preamble

Moode is my favourite way to listen to my digital audio library, not least because it has Camilla DSP built in, on which I depend to provide a parametric EQ correction curve to account for the tiny size of my listening room. Unfortunately my CD player does not benefit from this EQ since it is wired direct to my DAC. I wondered if I could connect my CD player to the Allo USBridge Signature digital streamer that runs Moode? It is based on a Raspberry Pi, after all, which is the ultimate in extensible computing devices, thanks to its HAT physical interface. I did some research and found a couple of Raspberry Pi HATs that offer S/PDIF input, and decided to go with the HiFiBerry Digi+ I/O.

I posted on the Moode Audio forums to see whether others had tried this, and the rather wonderful author of Moode pointed me to this rather excellent HOWTO from a user named Bitlab. My blog post is heavily based on his, but has some additions concerning getting the card recognized in the first place and the playlist configuration near the end.

Getting the S/PDIF input card’s driver loaded

I plugged it into the HAT connector, rebooted, then ssh’ed in to the rPi. Is the new device auto-detected?:

pi@moode:~ $ arecord -l
**** List of CAPTURE Hardware Devices ****

No, it is not. I found a guide at the HiFiBerry site to enable the driver for the HiFiBerry Digi+ I/O. The only step I did NOT follow was disabling the onboard rPi sound (i.e. I did not comment out the line dtparam=audio=on) else I found that Moode Audio would hang on boot.

So, is the card is now recognized:

pi@moode:~ $ arecord -l
**** List of CAPTURE Hardware Devices ****
card 2: sndrpihifiberry [snd_rpi_hifiberry_digi], device 0: HifiBerry Digi HiFi wm8804-spdif-0 [HifiBerry Digi HiFi wm8804-spdif-0]
Subdevices: 1/1
Subdevice #0: subdevice #0

Success!

Configuring ALSA to use the card

Note the card, subdevice and subdevice parameter values (2,0,0 in my case), then get more details about this input device :

pi@moode:~ $ arecord -D hw:2,0,0 --dump-hw-params

Recording WAVE 'stdin' : Unsigned 8 bit, Rate 8000 Hz, Mono
HW Params of device "hw:2,0,0":
--------------------
ACCESS: MMAP_INTERLEAVED RW_INTERLEAVED
FORMAT: S16_LE S24_LE
SUBFORMAT: STD
SAMPLE_BITS: [16 32]
FRAME_BITS: [32 64]
CHANNELS: 2
RATE: [32000 192000]
PERIOD_TIME: (10 2048000]
PERIOD_SIZE: [2 65536]
PERIOD_BYTES: [16 524288]
PERIODS: [2 65536]
BUFFER_TIME: (20 4096000]
BUFFER_SIZE: [4 131072]
BUFFER_BYTES: [16 524288]
TICK_TIME: ALL
--------------------
arecord: set_params:1343: Sample format non available
Available formats:
- S16_LE
- S24_LE

We can use that information to set up an ALSA configuration file so that the system has means to address the card and input using convenient labels. I created /etc/alsa/conf.d/hifiberry_digi+_inputs.conf with the following content:

pcm_slave.digi_in {
pcm "hw:2,0,0"
rate 44100
channels 2
}

pcm.SPDIF_input {
type dsnoop
ipc_key 12342
slave digi_in
bindings.0 0
bindings.1 1
}

In the pcm_slave part above, I name my device digi_in then I use the same 2,0,0 card reference from above; you’d change that for your situation. 44100 as the sample rate since in my setup I’ll only be using it to play back CDs. 2 channels because that’s how many my device has.

In the pcm part, I name the input SPDIF_input and the slave parameter is the name of the device from above. See here for an explanation of the other parameter values, plus more generic instructions if your device has a larger number of channels.

Testing the card

To test whether this newly-configured input works let’s try and record from it. I plugged my CD player into the electrical digital input (on the HiFiBerry Digi+ I/O, inputs are the TOSLINK socket with the black dust cover and the RCA jack to its right), started a CD playing, then issued the following command:

pi@moode: $ sudo arecord -D SPDIF_input -f S16_LE -r 44100 -c 2 test.wav
Recording WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo

Here we’ve referenced the ALSA input defined above and once again specified CD audio format options. Ctrl-C will stop the recording. ls -l to check that test.wav is longer than 44 bytes (i.e. that there are actual samples in it beyond the file header info), then the following command to play it back:

pi@moode: $ aplay -D plughw:1,0 test.wav

This time the card reference differs since my playback device is separate from the SPDIF input card. I heard my recording play back. Success again!

Making the card available to the Moode UI

Next, this input needs to be made available to Moode in some way so as to control things via the UI and also to benefit from the room correction EQ curve. Cunningly, this can be set up using a playlist! M3U playlist files support hyperlinks, and MPD (the audio player within Moode) supports an alsa:// link format to play back directly from an ALSA source. I created /var/lib/mpd/playlists/CD Player.m3u, containing the following:

#ETM3U

#EXTINF:-1, S/PDIF Input
alsa://hw:2,0?format=44100:16:2

Note that this refused to work if I tried to use the identifiers I set up in the ALSA config above – I had to go back to the numeric references. Also, the EXTINF line is ignored so technically isn’t needed.

In the Moode UI, I navigated to the Playlists and hit the refresh icon. My new playlist was displayed, so I clicked the three dots then Clear/Play. Pressed play on the CD player and voilà, I was basking in newly-EQ’ed compact disc sound :)

Bonus point

It is possible to edit the playlist via the UI and upload an image file to make it look a little more integrated.

Drawbacks?

As noted in this HiFiBerry article, and also explained to me by IanCanada (who has made quite a name for himself with his no-compromise rPi audio solutions), the one probable drawback with this solution is jitter. Simply put, jitter is the difficulty in identifying when each bit of a digital stream begins and ends. In systems where the clock signal is sent separately from the audio stream, this is solved. In systems where the S/PDIF is connected directly to a good quality DAC, this is also solved since good DACs tend to ‘reclock’ the incoming stream in order to recover the correct clocking. There also exist ‘reclocker’ rPI boards that can recover the clock directly from the audio stream, but the kicker is that it is apparently not possible to pass the resulting stream from these to the rPi I2S bus, which is the only way to pass the audio to the operating system running on the rPi.

So then, I will perform some listening tests and report back on whether I can hear a loss in quality separate from the EQ improvements that were the whole purpose of this guide!

Sending external email from TrueNAS FreeBSD jails

18 Apr 2023 at 09:02

admin

FreeBSD, FreeNAS, TrueNAS Core

No Comments

Background

The thing about FreeBSD jails and VMs is that they’re so easy to set up one tends to end up with many of them, each with its own purpose. Though they don’t have a physical form they’re still full servers, doing all the things servers do unless you configure them not to, and sometimes things can break. If one doesn’t have some way in which to keep tabs on them, such breakage can be subtle and may not become obvious until it is too late. So then, how to monitor a server’s ongoing behaviour? Well, this is a solved problem; FreeBSD, like all Unices, has perfectly good scheduling and logging features built in, and the results of scheduled jobs can trivially be configured to be sent to the server’s root email account. What is not so trivial is how to have such locally-sent mails continue to work whilst also having them forwarded to an account that it outside that server, such as your own personal email account. The main challenges are:

  1. Configuring the default email utility, sendmail, is witheringly complex, so setting it up to send mail locally and externally is non-trivial;
  2. The naive optimism of the original SMTP protocol allows one all too easily to arrive at a working but insecure setup that would leave the server open to attack or abuse; and
  3. Because of (2) many email services treat incoming mail with great suspicion, so it is becoming harder to convince them that your mail is legitimate.

My chosen solution to this conundrum is to leave sendmail doing just the within-server email, and then set up an entirely separate utility (SSMTP) to handle external mail forwarding. Yes, sendmail could do both, but I do not have the time or inclination to work out how to have it perform both within-server and outgoing email in a secure manner, so this is my way to compartmentalize risk. Note that there are many guides out there that will help you set up SSMTP to replace sendmail, but in my experience this breaks within-server email since SSMTP isn’t designed for that use case – it can literally only send mail via SMTP.

Installing SSMTP

pkg install ssmtp

Basic Configuration

Edit /usr/local/etc/ssmtp/ssmtp.conf as follows:

mailhub=<your external SMTP server>:<port used - one of 25, 465 or 587>

Note that this SMTP server is the one which is going to forward your mails, not that of the destination email address. Though it does not necessarily have to match the domain of the destination email address, it is probably more likely to work if they are the same. Various articles suggest that Gmail is not a good bet – they do too much behavioral checking and might not reliably forward your mails.

rewriteDomain=<your local domain name>

In order to improve your chances of having your mail accepted by the chosen forwarder, it is good to have it appear to come from a genuine domain. I use a domain that I set up with dyndns; there are many such free domain forwarder services. Note though that this domain won’t actually be used to route the mail in any way.

FromLineOverride=NO

This is the default value of this setting, but I think it is worth explaining it. The sense is, in my view, backwards. ‘No’ means that the From: line of the incoming mail will not override SSMTP’s ability to rewrite that value. In other words, ‘No’ means that SSMTP is allowed to alter the From: line. We very much want SSMTP to do so in order to improve the apparent validity of the mail; we’ll use the revaliases configuration later on to spoof the From address.

UseTLS=Yes

Presumably you want your SMTP communication with the forwarder to be encrypted?

AuthUser=<your username at the SMTP server>
AuthPass=<your password at the SMTP server>

Obvious. You ought to set the file permissions and flags for this file to be appropriately restrictive since these are clear-text values.

Spoofing the source address

SSMTP allows us to rewrite the From address of mails, as noted above. The mapping is done in the revaliases file, so edit /usr/local/etc/ssmtp/revaliases as follows:

mailnull:<hostname of server>@<value of rewriteDomain set above>

FreeBSD’s periodic utility, which is responsible for scheduled script runs, uses the username mailnull. Since you’re likely to have multiple servers you set up with this configuration, using the hostname of the server is a good way to distinguish between similar mails coming in from different servers. Finally the domain name should match the one set up earlier as rewriteDomain in ssmtp.conf.

Configuring root to forward its email externally

So far so standard; here comes the hack :)

BSD (and no doubt all other Unices) supports a .forward file, stored in each user’s home directory, to be used to forward mail sent to that user elsewhere. I learnt from its manpage that as well as containing literal email addresses, lines enclosed in double quotes will be executed as commands. This means I could insert a line in root’s .forward file in the following format:

"|/usr/local/sbin/ssmtp -F"<server hostname> root" <destination email address>"

Breaking it down, this command pipes each mail sent to root to the ssmtp command, the -F parameter allows me to modify the Sender line so that I know from which server the mail is coming, and sends it to the address specified. Remember that it is probably helpful if the destination address is at the same server you’re using for SSMTP’s mailhub value, though it is not mandatory.

Coda

Since setting this up, I’ve learnt that /etc/mail/aliases also supports the line format described for .forward above, so one could equally insert a line in that file instead, which is arguably a more central configuration file for a server’s email instead of .forward.

Configuring sendmail on TrueNAS FreeBSD jails

27 Mar 2023 at 21:45

admin

FreeBSD, FreeNAS, TrueNAS Core

No Comments

N.B. This post is as much an aide-memoire to myself as anything else. I found it difficult to find the right web search terms to use to learn how to configure sendmail, so this information was more or less hard-won by experience, hence I don’t want to forget it.

 

Basics

Sendmail’s configuration files live at /etc/mail. They are named sendmail.cf and submit.cf. They are not designed to be updated directly however – they are compiled by running the ‘human-readable’ versions (suffixed .mc instead of .cf) through a pre-processor, which is achieved by executing make whilst in the /etc/mail directory. This is where things get weird – sendmail.mc and submit.mc do not exist. Instead, a newly-installed jail will have the following files:

freebsd.mc

freebsd.cf (exact match for sendmail.cf)

freebsd.submit.mc

freebsd.submit.cf (exact match for submit.cf)

Making Configuration Changes

What changes you need to make are outside the scope of this blog, but the files to which changes should be applied are freebsd.mc and freebsd.submit.mc.

Applying Configuration Changes

Once those files are saved, execute make whilst /etc/mail is you current working directory. It is important to note that all ‘negative’ output, even warnings, is likely to cause the process to fail, so take heed of warnings and errors and work out how to fix them in the .mc files. Note also that the make process invokes sendmail using the previous set of configuration files, so your old sendmail.cf and submit.cf files must be in place before running make. The output of a successful make will be the same set of four files as listed in Basics above, except the file prefix will be your jail’s hostname instead of freebsd. The final step to apply this new configuration is to copy <hostname>.cf and <hostname>.submit.cf to sendmail.cf and submit.cf. I would recommend first saving the old files somewhere as backups, given that you always need to have a ‘known good’ set available in order to be able to make new configurations.

FreBSD shell setup basics (e.g. Ctrl-left arrow writes ;5D)

12 Mar 2023 at 16:57

admin

FreeBSD

No Comments

Configuring keyboard navigation

Binding keyboard keypresses to certain non-printing-character functions, such as cursor keys, can be done in multiple ways. This is the *nix way of course, but it doesn’t help if you’re in a server shell, editing your command prompt, and Ctrl-back barfs characters instead of moving your cursor.

Usually FreeBSD uses csh as it’s default shell. The necessary bindings can be added in ~/.cshrc:

bindkey "^W" backward-delete-word
bindkey -k up history-search-backward
bindkey -k down history-search-forward
bindkey '\e[3~'         delete-char
bindkey '\e[1;5C'       forward-word
bindkey '\e[1;5D'       backward-word

Change the default editor

Though most sources I find tell me that FreeBSD uses csh as it’s default shell, under some circumstances jails I create under TrueNAS Core use sh instead. To change to csh, use the chsh command. Having come from Linux I am accustomed to using nano to edit text files, whereas FreeBSD’s default editor is vi. It uses the default editor for some shell built-in functions (such as chsh), so I alter it by editing ~/.profile and changing the EDITOR shell variable to nano then logging out/back in.

Creating a guest WiFi network using a VLAN with a Cisco Aironet access point and OPNsense firewall

01 Oct 2022 at 16:21

admin

Networking, Privacy

No Comments

The aim of this howto is to explain the steps involved in setting up a second WiFi network in your home that does not have any access to all other devices within the home, such that your guests can have access to the internet but no access to your home network. I wrote it mainly because Cisco Aironet access points have a web interface that is very broken, causing me many dead ends and frustrated “Why isn’t it working? It should be working!” moments until I gave up and configured it via its console interface. The commands used are arcane if you are not well-versed in Cisco IOS so I wanted to write it all down whilst it was still fresh in my memory. I should also note that this guide assumes IPv4 only – I do not use IPv6.

The process breaks down into three sections :

  1. Configure the Cisco Aironet access point WiFi networks and VLANs.
  2. Configure your switch(es) to pass the VLAN traffic.
  3. Configure the OPNsense firewall to route the VLAN traffic appropriately.

1. Configuring the Cisco Aironet access point WiFi networks and VLANs

I will assume that you already have a cable that converts the RJ-45 serial connector on the AP to something your computer can understand (old-skool serial or perhaps a serial-to-USB convertor). Such things are easily found online. I am configuring from a Linux machine; YMMV if you’re using Windows or a Mac, but both have equivalent programs to open a console on a serial connection. Under Linux the serial-to-USB adapter automagically creates a device entry in /dev, so I used screen to open a console connection, as follows:

screen /dev/ttyUSB0

Cisco IOS configuration operates using the concept of shells-within-shells. To first of all get into the ‘root shell’ from the initial shell we type:

en

Enter your root password if one is configured, when prompted. N.B. This password can be adjusted via the web interface – it is the one for the ‘cisco’ user. From here, the command to display the current configuration is:

show run

In terms of altering configuration, one either has to alter one of the existing lines or enter a prefix of ‘no’ to disable that item. IOS will simply throw an error and make no change if you get the syntax wrong. It is also important to bear in mind that one only alters the current running configuration, and a specific command is required to write that new config to NVRAM so that it survives a reboot of the device.

To start configuring, we need to enter the configuration shell with the command configure terminal, which can be abbreviated as follows:

conf t

We have the following items to configure, and due to the vagaries of Cisco APs they have to be done in this order:

  1. Define VLANs
  2. Define SSIDs, associating each with a VLAN
  3. Associate the SSIDs with a radio

Define VLANs

The way VLANs work, there is always a so-called ‘native’ VLAN which is that used by any traffic that is not explicitly tagged to use a VLAN. By convention, the native VLAN is number 1. We need therefore to explicitly name the native VLAN and create the one we’ll use for guest WiFi traffic:

dot11 vlan-name Native-VLAN vlan 1
dot11 vlan-name Guest-WiFi vlan 2

Define SSIDs

Here’s the syntax to define two SSIDs (your personal one and the one to be used for guest WiFi):

dot11 ssid personal ssid name
vlan 1
authentication open
authentication key-management wpa version 2
guest-mode
wpa-psk ascii personal wpa2 password

dot11 ssid guest ssid name
vlan 2
authentication open
authentication key-management wpa version 2
guest-mode
wpa-psk ascii guest wpa2 password

Obviously enough, the items in italics are for you to supply and cannot contain spaces. For those who’re interested, authentication open means that authentication is handled by the AP itself (i.e. not a captive portal or RADIUS or whatever) and guest-mode, bizarrely enough, means broadcast the SSID.

Associate SSIDs

We now associate the VLAN’ed SSIDs with a radio. For a dual band device the simplest approach is one network per radio. Cisco APs do support multiple SSIDs per radio, but I couldn’t work out how to configure it! Let me know if you have a good resource for this. Each time we configure an interface IOS drops us into a new shell, so we have to exit the sub-shell each time. Doing so also causes the radio configuration to be applied and the radio restarted. Dot11radio0 is the 2.4GHz radio, whilst 1 is the 5GHz. Channel can be set to your preference for radio0, whilst 5GHz auto-selects the least congested channel automatically.

int dot11radio0
ssid personal ssid name
channel 2422
encryption vlan 1 mode ciphers aes-ccm

exit

int dot11radio1
ssid guest ssid name
encryption vlan 2 mode ciphers aes-ccm

exit

Save configuration changes

With all configuration done we need to write it to the EEPROM so that it will survive a reboot, as follows:

wr

Now you can disconnect from the access point.

Configure your switch(es) to pass the VLAN traffic

This item will depend upon the switch topology you have and the models of device you use. The rules to follow are:

  • Ensure you’re using 802.1Q tagging and not port-based tagging.
  • Ensure VLAN 1 is assigned to all ports and is untagged.
  • Assign VLAN 2 as tagged to the port into which the Cisco AP is plugged.
  • Assign VLAN 2 as tagged to the port into which your firewall (or router) is plugged.

In this way, VLANs 1 & 2 can pass traffic from the AP, 1 traffic can go anywhere, 2 traffic can only go to and from the firewall. In truth, VLAN 2 can be tagged to all ports if you’d like to avoid forgetting that certain ports are ‘special’; packets tagged as being for VLAN 2 by the AP are going to ‘travel down’ the second VLAN and end up treated uniquely by the firewall. The trade-off of this approach is that any device physically connected to the switch can snoop on VLAN 2. In this application that’s not really an issue – our desire is to prevent VLAN 2 traffic from entering VLAN 1.

Configure the OPNsense firewall to route the guest VLAN traffic appropriately

Okay, your guest WiFi VLAN traffic is now battering itself against the LAN-side port of your firewall, so you need to configure the firewall to know what to do with it. Setting up OPNsense breaks down into:

  1. Define the existence of the VLAN.
  2. Create a virtual interface attached to the VLAN.
  3. Enable DHCP for the virtual interface.
  4. Create firewall rules to prevent the virtual interface seeing local networks but allowed out on the internet.

Define the existence of the VLAN

I followed this guide, which breaks down to going through the following menu path:

Interfaces > Other Types > VLAN > Add

The VLAN tag will be 2 (if you used the same as I did above) and the Parent Interface will be that of your LAN. Give it a meaningful name and leave the VLAN priority as-is.

Create a virtual interface attached to the VLAN

Follow this menu path:

Interfaces > Assignments

Then beside ‘New interface:’ at the bottom of the list select the VLAN name that you created in the previous step and click the + button.

It will now be on the list of interfaces in the menu tree on the left. Select the new interface, check the Enable box, choose a subnet that devices in the VLAN will use and enter it in the IPv4 Address (e.g. 192.168.100.1/24) and save.

Enable DHCP for the virtual interface

Follow this menu path:

Services > DHCPv4 > name_of_your_interface

Check the ‘Enable’ checkbox and enter a range of addresses that will be handed out to devices that join the guest WiFi network. Obviously these will need to be in the subnet you defined for the interface in the previous step.

Create firewall rules

Being a guest WiFi network, the usual idea is to allow guest devices to the internet but not to any device on the local network. With firewall rules there are many ways to skin a cat, but I initially used the guidance in this OPNsense forum post, which prevents access to any private address ranges (that is, those defined in RFC1918 – 192.168.0.0/16, 172.16.0.0/12 & 10.0.0.0/8). This turned out to be over-strict, since it prevented traffic flowing even within the Guest WiFi subnet. I therefore listed my private LAN’s range specifically instead. Create a rules alias named that covers these ranges:

  • 192.168.<your normal LAN>.0/16
  • 172.16.0.0/12
  • 10.0.0.0/8

Then, add the following two rules at the end of the rules list for the guest WiFi interface:

  • block destination alias_defined_above
  • allow destination all

These two rules in this order mean that any traffic that tries to reach any private address range via the firewall will be blocked, but any other traffic will flow (i.e. internet traffic). Traffic from one device to another within the guest WiFi subnet will flow, since this setup only blocks traffic that arrives at the firewall, which will only happen if the packet cannot find it’s destination within its source subnet.

← Older Entries