[Technology] Accessing files.web.uoguelph.ca from Linux when off campus

Until a few weeks ago, I was able to login remotely to my University's web hosting server.  Now, I need to connect via VPN first.

On Fedora Linux, I made sure I had vpnc installed and NetworkManager-vpnc.   Then I went to Network Manager's settings/preferences, added a new connection (+ in the bottom left), chose VNC, and chose the vpnc client.  Then I used the settings from this page:

https://www.uoguelph.ca/ccs/internet-phones/virtual-private-network-vpn/using-vpn

It beats using the creepy script from vpn.uoguelph.ca at least.

[General] Heading Homeward

It's the Flesherton Fling this weekend!  You should totally come out.  We used to have the Split Rail Festival.  Then we started making fences from metal and it seemed outmoded.  Now we just have flings - the moral decay of our century.

[Technology] Automating my life away

A decade ago, part of a job was to help automate creation of a report that normally involved a human scanning through dozens of spreadsheets each year, copying and pasting numbers and generating graphs and tables, manually.

The place where I worked predominantly used Perl.  (It was a science laboratory.)   My solution was to use OpenOffice's GUI to bulk-convert the Microsoft Excel spreadsheets (binary XLS files) to whichever file format OpenOffice used at the time that was actually a collection of XML in a zipped directory. (Was that .sdc?)

I then used Perl's libxml wrapper (probably wrapping libxml2) to parse the XML into an SQLite database.  (Or was it PostgeSQL like everything else there?)  I had tried to use some other Perl XML library, but it was Super Slow compared to the snappy libxml2 (written in C).  Like, a difference in many minutes to run versus one second.

From there, I wrote some Perl CGI scripts to automatically generate most of the tables and graphics into HTML form, in some fashion that looked nice when imported into OpenOffice. (Really?!)  The person who was responsible for the report, who normally took a few weeks to do, said that it would save him so much of the time, that it would take him now a couple days to look at the data and write conclusions and maybe do some cleaning.  He marveled at automation and how quickly he could be replaced.  The automation revolution in action.

Today I used LibreOffice to convert about a hundred spreadsheets into CSV files with this one command:

find . -name "*.xls" | while read line; do
  (cd "$(dirname "$line")" &&
   libreoffice --headless --convert-to csv "$(basename "$line")");
done

I am not sure that OpenOffice could do headless conversion back then.  I was probably overwhelmed by all the potential everywhere to discover the functionality if it exists.  This would have helped free a few hours of my life.

Automation, a revolution.

UPDATE:
Converting to CSV only does the 1st sheet tab.  You apparently can't specify a different sheet to be the active sheet.  I also had to convert some other old spreadsheets for easy parsing (mostly just grep'ing through them to find data).   In that case, I used --convert-to html which was beautiful, even if one of the resultant HTML files was 138MB large.

[Technology] Starting Firefox Faster

summary: if you use pinned tabs, and they're all loading at start, despite setting tabs to not load until selected, this post is for you.

I have two hard disks, a 24GB SSD (fast!) and a spinner.  I noticed that to fully load Firefox and wait for everything to settle, it was taking about a minute and a half, back when my root directory was on the HDD.  Recently (see earlier posts), I moved my root onto the SSD.  That improved things marginally.  Firefox was now settling in around 1m12s.  I tried moving my profile onto the SSD.  Now I was down around 1m00s.  (These are averaged from multiple warm runs.)  That is still abysmal.

One thing I notice is that a private web app I wrote (look up My Daily in past posts maybe) is guarded by an .htaccess file, and I have it as a pinned tab.  In that full minute, before things finish settling, I get prompted to authenticate myself.  I then recalled that I had enabled the Firefox option to load tabs when clicked (Preferences > General > "Don't load tabs until selected"), so why is it loading?

All the pinned tabs actually seem to be loading at startup.  Grr.  Perhaps that is tunable?

I went to about:config, and sure enough, typing in "pinned" filtered down to this setting:

browser.sessionstore.restore_pinned_tabs_on_demand

defaulting to "false".  I set it to "true", and now Firefox takes <3s to open and settle.  Sure, clicking on a pinned tab now takes a normal time to load the tab, but it's better than loading GMail, Google Calendar, Blogger, Google Drive, Google Keep and my web app ALL AT ONCE.

[Technology] Resizing my encrypted /home partition

I just wrote a post about why I reinstalled Fedora 22: because I wanted root to be on the SSD, and I needed to purge a lot of old installed software anyway. :P  And how that led to trying out the open source b43 wireless driver.

This is more interesting than my wireless problem.  Because I no longer needed a / partition on my HDD, I wanted to extend my home partition.  It isn't as dead-simple as back a decade ago.  Back then, I would use fdisk to delete and recreate partitions and resize2fs to resize the file system.

Now I have layers upon layers of useful abstraction.  I'm calling each something potentially inaccurate, and the tool I needed in brackets. (Those with wildcards match multiple tools with the same beginning.)

Top

• ext4 file system (resize2fs)
• Logical volume (lv*)
• Volume group (vg*)
• Physical volume (pv*)
• partitions (fdisk)

Bottom

At first I was almost a little annoyed at the apparent complexity, but it's actually straight forward.  My drive is still divided into multiple partitions in a partition table.  (/dev/sda1, /dev/sda2, /dev/sda3, etc; I have 8 for various reasons).

Physical volumes seem to correspond to those partitions, but mostly just the useful storage ones (e.g. I don't have them for EFI or swap partitions).  pvcreate lets me define them.

The physical volumes are collected into volume groups.  For example, /dev/sdb1 (my intended root) might be a part of "pantaloons-fedora-ssd" (if I named my machine pantaloons), and /dev/sda2 and /dev/sda4 might both be part of "pantaloons-fedora-hdd" (two large areas for my /home that I'd like to combine later, but which are currently separated by some other, annoying partition).

The volume group kind of looks like a single contiguous (!) entity (despite being a collection of disparate physical partitions).  Then, I can define logical volumes atop the volume groups.

I can have both / and swap as separate logical volumes atop "pantaloons-fedora-ssd" (talk about cramped - maybe I don't want to hibernate after all), and have a single logical volume for /home atop the "pantaloons-fedora-hdd" volume group.

This is where things get a little trippy though.  Where does the encryption happen?

For my root file system, it looks like it happens like:
partition /dev/sdb1 > LUKS > physical volume > EXT4 file system

For my home file system, it looks like this:
partitions /dev/sda2,/dev/sda4 > two physical volumes > 1 volume group > 1 logical volume > LUKS > EXT4 file system

FS	Ext4		Ext4
(LUKS)			/dev/mapper/luks-BAR
LVs	root	swap	/home
VGs	pantaloons-fedora-ssd		pantaloons-fedora-hdd
PVs	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
(LUKS)	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
partitions	/dev/sdb3
disks	/dev/sdb (SSD)		/dev/sda (HDD)

The home partition and its LUKS (using cryptsetup/dm-crypt) was originally configured through whichever tool Fedora provided years ago (preupgrade? fedup? anaconda?), while the configuration on my SSD was what was recommended in F22.

Basically, I had to (with steps that actually increase space emboldened):

• define a new partition in the partition table, 
• define a new Physical Volume (PV) over top it, 
• extend my Volume Group (VG) to include the new Physical Volume
• unmount the file system I was going to extend, 
• close its LUKS setup
• extend the file system's Logical Volume (LV) overtop the new space in the Volume Group
• reopen the LUKS setup
• resize the LUKS setup (are these two out of order? hope not!), and
• resize the Ext4 partition

A bit more detail, here are the steps I used on my system (from my bash_history, probably missing some steps; I think I also used lvremove or pvremove to remove LVs and PVs related to the old home partition, oh well!):

• # cfdisk, to delete the old root on the /dev/sda4 partition and recreate it
• # pvcreate /dev/sda4 (creates a physical volume)
• # vgextend pantaloons-fedora-hdd /dev/sda4 (add the PV into the VG increasing its overall size)
• make sure /home isn't mounted
• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (let's make sure the inodes are fine on our file system before messing with it)
• # cryptsetup luksClose luks-BAR (close LUKS while we change the underlying LV)
• # lvextend -L +61G /dev/pantaloons-fedora-hdd/home (increase the size of our LV within the VG)
• # lvextend -L +242M /dev/pantaloons-fedora-hdd/home (catch a little more space)
• there is probably a way to just extend it to the full space available, maybe read the man page
• # cryptsetup luksOpen /dev/pantaloons-fedora-home luks-BAR
• # cryptsetup resize luks-BAR (resizing the LUKS setup)
• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (for paranoia maybe)
• # resize2fs /dev/mapper/luks-BAR (finally, let's resize our Ext4 file system)

Sadly, I can't cite the sources I pieced some parts together from, because I did this form a live USB key which is supposed to keep history as persistent storage but the browser history was empty when I tried checking it later :S  However, man pages were also useful.

[Technology] How to Mount the Overlay Filesystem of a Live USB

So, I wanted to demonstrate the commands in the previous post used for resizing my file system, but of course I did that from a live USB (since I wanted my file systems unmounted). However, I did not want to reboot my computer just now to copy root's bash history, so I looked up how to mount the overlay using these instructions:

https://fedoraproject.org/wiki/LiveOS_image

The simplified version would be:

1. insert USB key into computer
2. identify it
   $ journalctl --system
   mine was created as /dev/sdc1
3. mount it, creating target directories as you go:
   # mount /dev/sdc1 /tmp/sdc1
4. mount the read-only squashed file system:
   # mount /tmp/sdc1/LiveOS/squashfs.img /tmp/squashfs
5. create a block device for its ext3fs image:
   # losetup /dev/loop1 /tmp/squashfs/LiveOS/ext3fs.img -r
6. create a block device for the overlay:
   # losetup /dev/loop2 /tmp/sdc1/LiveOS/overlay-LIVE-foo
7. calculate size of loop1
   # blockdev -q --getsz /dev/loop1mine was 12582912
   
8. define a device combining the two
   # dmsetup create myliveos --table "0 12582912 snapshot 7:1 7:2 P 8"
   myliveos: my own name for the device, call it what you will
   12582912: the # of blocks I acquired from step 7, put in yours
   7:1 references loop1
   7:2 references loop2
9. mount the combined device
   # mount /dev/mapper/myliveos /tmp/myliveos

Now I can cd into /tmp/myliveos, and get my .bash_history!

[Technology] Installing Fedora 22, again: wireless Internet and resizing file systems

I have a 24GB SSD and a 500GB HDD in my computer.

A few months ago, every time my computer hibernated or slept, my file system on my SSD got corrupted, and I had to fsck it to repair it.  I ended up just creating a new partition on my HDD and installing my OS there.

Recently, I've grown reckless, and decided I wanted to have my OS on the SSD after all.  I did some SMART tests and my SSD seems to be fine, so pretending to forget about the earlier drama, I reformatted the SSD and re-installed Fedora 22.

Wireless Internet

Of course, the Internet doesn't work out of the box.  That's because of Broadcom's proprietary drivers and firmware.  Previously, I used the wl driver, using akmod-wl and kmod-wl.  My device:

03:00.0 Network controller: Broadcom Corporation BCM43228 802.11a/b/g/n

I looked at the open source driver, b43, though, and it claimed to now support it(!).  I apparently needed the firmware from the Windows driver, though.  Sigh.  Indeed, journalctl --system (instead of dmesg) reported this:

[  162.191310] b43-phy0: Broadcom 43228 WLAN found (core revision 30)
[  162.191764] b43-phy0: Found PHY: Analog 9, Type 4 (N), Revision 16
[  162.191776] b43-phy0: Found Radio: Manuf 0x17F, ID 0x2057, Revision 9, Version 1
[  162.192544] Broadcom 43xx driver loaded [ Features: PMNLS ]
[  162.192570] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2
[  162.192596] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2
[  162.192995] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2
[  162.193015] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2
[  162.193019] b43-phy0 ERROR: Firmware file "b43/ucode30_mimo.fw" not found
[  162.193022] b43-phy0 ERROR: Firmware file "b43-open/ucode30_mimo.fw" not found
[  162.193024] b43-phy0 ERROR: You must go to http://wireless.kernel.org/en/users/Drivers/b43#devicefirmware and download the correct firmware for this driver version. Please carefully read all instructions on this website.

This lead me to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/#Fedora, which has a broken link to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43#other_distros further down the page.


In the end, I had to do these things

$ dnf install b43-fwcutter wget     # dnf instead of yum
$ export FIRMWARE_INSTALL_DIR="/lib/firmware"
$ wget http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2
$ tar xjf broadcom-wl-5.100.138.tar.bz2
$ sudo b43-fwcutter -w "$FIRMWARE_INSTALL_DIR" broadcom-wl-5.100.138/linux/wl_apsta.o
$ modprobe b43

And it worked.

Mounting encrypted /home partition

I cheated and used gnome-disks to make my encrypted /home partition auto-mount.  Sorry.

[Microblog] Transient Thought

Bitcoon, a Canadian fork of bitcoin?

[Microblog] Transient Thought

I have lost my playfulness.  I am grown too serious.

[General] Simplicity and Innovation

It can be so hard to just find something simple/"normal".

I go into a store and I want to buy a binder.  They all have designs on them.  None of them are just "normal" - plain and simple.

I want to find a simple muffin recipe.  No, no walnuts, no blueberries, no bananas, absolutely no animal parts, no no no no.  Just a muffin.  Recipe websites have large categories with 90 different types of muffins.  I don't care.  I just want some fat, some sugar, and some wheat in the shape of a muffin.  I'll add something later.

Perhaps that will be where AI assistants/future websites have room to grow: helping simplify what should be simple.

Eventually I found this:
http://vegweb.com/recipes/simple-muffins