Unix error detection – cksum, md5sum, and shasum commands

This post will be a tutorial on Unix error checking. These commands are available in all Unix systems (that I have tested), though they have slightly different forms in each one.

Error detection codes are used to detect errors caused by either disturbances in a noisy communication channel, or deliberate modification by malicious users. There are many different methods of error checking. One form is called a checksum or hash. This is when some operation with even probability distribution is applied to the bytes, blocks, whatever, of a file at both ends of the channel to produce a much shorter value, called the sum. The sums at both ends are compared (usually by humans rather than by computers) to see if they match. If they don’t match, something has gone wrong in the transmission, and the values need to be retransmitted.

The first error detection command we will look at is the cksum command. cksum produces a checksum of a file by computing a remainder of a polynomial division of its contents. It then prints the checksum and the size of the file side-by-side.

bash-3.2$ cksum hackers.png
1967572667 13865 hackers.png

Here the first value is the checksum and the second value is the size of the file, followed of course by the name of the file.

We now compare these values to the values supplied by the site we downloaded the file from. If they match, we know the file probably downloaded correctly (it’s not an absolute guarantee, but the probability of an error is now known to be negligible).

cksum is used because it is fast and simple to implement. It has a couple of problems, though. First, implementations may vary. When using cksum you have to make sure you and the supplier of the file are using the same CRC code.

The second problem is that while CRCs protect a file from accidental modification by noise; they do not protect it from deliberate modification by malicious users. This is because multiple files can have the same checksum, and a middleman can easily inject a fake file into the network that has the same checksum as the one we are trying to download, despite being different. To guard against this sort of attack, we need what is called a one-way hash, which is a hash that is computationally very difficult to reverse.

There are a couple of these algorithms that are in common use on Unix systems. One is the MD5 algorithm, and the other is the SHA series of algorithms (in Unix the default is SHA-1). Each of these message digest algorithms, as they are called, has at least two forms – the form used in BSD systems (including Mac OS X) and the form used in Linux. Here is a table showing each of them:

Algorithm Linux form BSD form
MD5 md5sum md5
SHA-1 sha1sum shasum

Let’s look at a sample command in Mac OS X:

bash-3.2$ md5 hackers.png
MD5 (hackers.png) = d28903b7e06cd169f5e4ff59be348fb6

Unlike the cksum output, which is in decimal, the MD5/SHA-1 output is in hexadecimal. It is also much longer. Again, we compare this value to the one given by the providers of the file to make sure it is correct.

EDIT (November 11, 2016): Man, I’ve been putting off correcting this for too long.  I said something really stupid in this post, which was that a one-way hash has only one file mapped onto each hash.  Obviously, this is mathematically impossible, because a one-to-one function can only exist between two sets of the same size.  I have corrected it now.  God, this is a cringeworthy mistake, and I’m totally embarrassed by it.

Running Arch Linux from a live ISO

Recently I decided to try another Linux distro – Arch Linux.  I wanted a distro that I could use to experiment with different boot loaders, different partition table setups, different filesystems, etc.  Because I didn’t want to use up all my hard drive space installing Slackware several times, I decided to use a more minimalist distro.  The first thing that came to mind was Arch.

Arch is in many ways similar to Slackware, but in other ways it is a polar opposite.  It is similar to Slackware in that it doesn’t default to a GUI environment and it expects users to be technically competent and know how to use the command line.  It is the opposite of Slackware because it does this for an entirely different reason.  Whereas Slackware aims at giving the user more options while being complete out-of-the-box, Arch Linux follows the KISS philosophy (that stands for Keep It Simple Stupid), and thus ships only with the minimum tools and interface needed to function and to be useful.  Thus, it has no graphics, because graphics take a lot of memory and processing power.  Arch Linux is the vi to Slackware’s Emacs.

I downloaded the ISO file for Arch Linux from the linaxe.net mirror, which can be found here, before running an MD5 checksum against it to make sure it downloaded properly. As usual, I created a virtual machine in Virtual Box and booted from the ISO. It turned out that you don’t actually have to install Arch Linux to use its full capabilities. It comes fairly complete as a live distro, which can be booted directly from the install disk.

Arch Linux Startup

I am liking Arch Linux so far. In a way it actually follows Slackware’s design philosophy better than Slackware has (for me anyway), because it comes complete with all the drivers I need for networking (and the ability to network makes an operating system ten times more useful than one that doesn’t have this ability).

As is usually the case, Arch uses its own special package manager. This one is called pacman. I haven’t figured out how to use it yet, and I can’t really use it while I’m booting it live because there’s nowhere to install the packages to (since the hard drive isn’t partitioned and the live ISO is read-only). This is one of the disadvantages of booting Arch Linux in live format, and I do plan to install it to the hard drive at some point, once I have read the manual for how to do so.

The top program in Arch Linux is more full-featured than in other operating systems. It lets you view things like the sizes of the code segments, stack/heap segments, and shared memory segments of the processes, and it allows you to change the colors. It also shows bar graphs of some of the system stats. In many ways, it’s a lot more like htop.



At the same time there are several things missing from Arch Linux, at least the live version.  There’s no mailx program, and there’s no cron daemon. I can kind of see why these programs would be missing in a live distro, since they involve storing information on the hard drive.

So I guess the question now is, will I install this OS to the hard drive? If so, when will I do it? Only time will tell.

A problem I’ve had with the X Window System in Slackware

Linux comes in many flavors. Some, like Ubuntu, are idiot-proof. They provide easy-to-use tools like the Synaptic Package Manager, GParted, etc. to accommodate those who are afraid of typing. Many Linux distros are set up so you can use them without ever having to touch the command line. Slackware is not one of these distros. And that’s why I like it.

Slackware is not graphical by default. It boots into the Bourne shell and gives you a prompt, with no indication of how to activate the GUI. You can of course activate the X Window System by using the startx command. I wanted to have some graphics on my system so that I could play with different desktop environments (I’ve seen some screenshots of Fluxbox and it looks amazing).

My Slackware installation has a problem, however. When I type startx, I get an error message saying that no screens were found and telling me to look at the log file /var/log/Xorg.0.log. Allow me to give you a visual of the log file in question:


It looks like I’m missing a couple of drivers.  One is fbdev, a loadable kernel module, and the other is the module corresponding to the device file /dev/dri/card0.  As is usually the case, I have no idea what these drivers do, so I proceeded to do some research on Google.  I didn’t find much in the way of documentation, just a lot of Linux forum threads started by people who had the same problem I did.

I don’t know what to do now.  I mean, I have a vague notion of what to do, but I don’t know where I can find these drivers.  Since my Slackware installation can’t access the network, I will have to get a specific URL (which I’ve inquired about on linuxquestions.com), then download the packages through Mac OS X, wrap them in an ISO file which I will then import into the Slackware VM, then use pkgtool to install the packages from a local directory.

Midnight Commander: A character-based menu interface for Linux

I mentioned Midnight Commander in my last post, and now I will go into greater detail about this program here. I really like Midnight Commander. I like it for three reasons: 1. It’s convenient, 2. It gives you a lot of options, and 3. It looks nice.

Here is a screenshot of Midnight Commander running in Slackware Linux:


Midnight Commander lets you change what is displayed in both the left and right columns.  For example, here is a screenshot of Midnight Commander running in Arch Linux, with the right column set to display file information:


Pretty nifty, huh?  It gets better, because the menus in the file manager allow you to do just about everything with the files that you can do from the command line.


I haven’t tried most of these yet, but judging from the menus, Midnight Commander appears to be a fairly powerful tool.

You can also use the -b switch to force Midnight Commander into monochrome mode:


This is useful if you prefer a black background to a blue background.

And of course there’s the document viewer:


And the text editor, here showing my .plan file:


The text editor, which itself can be invoked from the command line using the mcedit command, is a full-featured menu-based text editor complete with syntax highlighting.

Next up: a post about my struggles to get the X Window System to work in Slackware.

Exploring packages in pkgtool

I like to look through the package repositories in pkgtool, the Slackware package manager, to see what packages are installed in my Slackware installation. pkgtool provides a menu-driven interface for installing, updating, removing, and viewing packages. One of the (possibly unintended) effects of this is that it adds a level of discoverability to the Slackware command line.



I’ve found a lot of neat or useful commands by looking through packages on my system. I will list a few of them:

Midnight Commander (mc): An MDI for Unix-based systems, including a file manager, a document viewer, and a text editor.



bpe: The Binary Patch Editor, a hex editor for Linux.


sc: The Spreadsheet Calculator. Looks a lot like Visicalc, but it appears to have different commands. I just found it and have no idea how to use it yet. Will have to explore it some more.  For now I’ve just taken a screenshot of a blank spreadsheet, because I don’t know how to add data to it.


So far this is all I’ve used pkgtool for, at least directly. I haven’t used it to install packages, because I’m not entirely sure yet how to do that. Apparently you have to get packages from somewhere else and then install them from a local directory. I’m not sure. More to learn about, I guess.

The Slackware installation process

Recently I installed Slackware Linux.  In this blog post, I will detail the installation process I went through as well as my first impressions of the OS.

Slackware is a Linux distro that is supposed to be hard to install, hard to configure, and hard to use.  This is the main reason I wanted to install it – for the challenge and for geek cred.  Every time I engage in a major project like this, I learn something from the experience.  That’s why I love it.  This is especially true of the Slackware installation process, because Slackware makes you choose all of your packages and settings, and it also makes you partition the hard drive from the command line.  Now I was doing this whole installation without reading the guide, so I’m not sure if I did everything right, but I got the crucial parts of the operating system installed (still a few kinks to work out with graphics and networking, but otherwise it works fine).

The first step in the Slackware installation process is of course to download the ISO file, which can be found here. If you’re installing it directly to your computer, you can then burn the ISO to a DVD or Flash drive using a utility like dd. Since I was installing Slackware to a virtual machine, I skipped this step.

The next step is to insert the install disk (or ISO file) into your computer (or virtual machine) and boot from the disk. Then this screen appears:

Slackware 1

At this point I just typed Enter twice to skip the maintenance and keyboard selection steps.

Slackware 2

Then enter “root” to log in as root.

Slackware 3

The next step is to partition the drive.  There are four options at this stage: you can use either an MBR or a GPT partition table, and you can use either a full-screen, menu-driven program or a command-driven program.  The four programs are fdisk, cfdisk, gdisk, and cgdisk.

Here are is a screenshot of fdisk when you start it:

fdisk 1

And the commands for fdisk:

fdisk 2

Here is a screenshot of cfdisk, with three partitions set up:


Originally I had an 8 GB hard drive with 6 GB for the filesystem and 2 GB for the swap space, but I soon realized this was not enough, so I created a 20 GB hard drive.

When you partition the drive using an MBR scheme, you will have to decide what partition type to use for each partition.  A partition type is an 8-bit field in the MBR that denotes how the partition is used.  Here is the list of partition types as shown by fdisk:

fdisk 4

As you can see, a standard Linux partition has a partition type of 83h and a Linux swap partition has a partition type of 82h.

An explanation of the swap partition is probably in order, for those who are unfamiliar with Linux.  Basically, an operating system will run out of space in RAM and will swap pages of memory out to the hard drive.  Linux gives you two options for how this is done – either a swap file in the Linux partition, or a separate partition devoted to swapping.  The latter is generally more efficient.

Now we get to the fun part – actually installing Slackware.  In the setup directory in /usr/lib/setup you will see several executables.


The program you want to run is setup. Type setup at the command prompt. You will then see this screen:

VirtualBox_Slackware 15

Go down to “TARGET” and press the Enter key.  The Slackware installer will now format your target partition with a Linux filesystem of your choice:

VirtualBox_Slackware 17

VirtualBox_Slackware 19

As you can see, there are several filesystems to choose from.  I chose Ext4 just because I’m not familiar with the non-Ext filesystems.

After formatting, you will be prompted for what media you are installing Slackware from.  This part is pretty straightforward.

VirtualBox_Slackware 21

Then you will be prompted to choose which categories of packages to install.  Each category is explained pretty well, so I feel no need to explain them further:

VirtualBox_Slackware 22

Then you choose what installation mode to use:

VirtualBox_Slackware CLI_27_05_2016_10_44_21

I chose newbie mode the first time I installed it (I had to install it multiple times before I got it right), because I wanted to understand each of the packages that were being installed.  Newbie mode isn’t recommended, though, because it takes a really long time, especially if you’re installing the X environment.

Here is a sample screenshot of the kernel being installed:

newbie install 7

The Slackware installer doesn’t tell you this, but what it’s actually doing is running the pkgtool utility. This is a menu-based package manager that provides an easy installation process for Slackware packages. Part of the Slackware philosophy is that the operating system should be complete out-of-the-box. This means that the initial installation comes with a lot more software than, say, Debian. This makes Slackware preferable in my opinion, because searching the Internet for drivers and other packages is a pain in the ass.

The next step is to configure the system. The installer runs several config programs for you. You can run these programs again at any time if you want to change the configuration of your system.




When I finished with the installation and rebooted, I found that the Linux partition was unbootable.  I went to a Linux forum and asked about this problem, and they suggested using the chroot command to change the mount point for /dev/sda1 to my root. That worked.

There are still some problems with this installation that I haven’t worked out. There’s no network access. The X Window System can’t find my screen when I try to start it up. And the top utility makes it so my commands don’t echo. I will have to figure out how to solve these problems in the future.