Reverse engineering Apple’s in-terminal emojis

I found a somewhat interesting feature in the Apple terminal, supported by both Terminal.app and iTerm2. Essentially, it allows for the printing of emojis in the terminal. I discovered this when I noticed that the Homebrew package manager prints beer and tap emojis when it is downloading and installing packages. I was curious about how these emojis are represented, so I used a program that I had written, called hex, which essentially just displays the hex values of characters entered by the user. I’ve used this program to find escape sequences for things like function keys and the like, allowing me to implement these inputs in my programs.

Here is the source code for hex. It’s a fairly simple program…


 1 // This program displays the octal, hex, decimal,
 2 // and byte character representations of characters
 3 // typed at the terminal.
 4 
 5 #include <stdio.h>
 6 #include <termios.h>
 7 #include <signal.h>
 8 
 9 struct termios term;
10 struct termios save;
11 
12 void stop( int );
13 void cont( int );
14 
15 int mainint argc, char **argv ){
16         tcgetattr0, &term );
17         save = term;
18         term.c_lflag &= ~( ICANON | ECHO | ECHONL );
19         signalSIGINT, stop );
20         signalSIGTSTP, stop );
21         signalSIGCONT, cont );
22         tcsetattr0TCSANOW, &term );
23         // Section where the program executes
24         // in raw mode:
25         unsigned char c;
26         for(;;){
27                 c = (unsigned chargetchar();
28                 printf"%3o\t%2x\t%3d\t%c\n", c, c, c, c );
29         }
30         // End raw mode:
31         tcsetattr0TCSANOW, &save );
32         return 0;
33 }
34 
35 // Restores terminal settings before allowing
36 // the signal.
37 void stop( int signum ){
38         tcsetattr0TCSANOW, &save );
39         signal( signum, SIG_DFL );
40         raise( signum );
41 }
42 
43 // This function undoes the cleanup performed
44 // by stop() when the process is suspended.
45 void cont( int signum ){
46         tcsetattr0TCSANOW, &term );
47         signalSIGTSTP, stop );
48 }

I copied the beer mug emoji from the terminal after downloading a package. I then started the hex program and hit Command-V, then copied the hex bytes (there were four of them) by hand into a file using a hex editor. To test it, I ran cat on the file I had created, and lo and behold – a beer emoji was printed to my terminal.

I then decided to test a concept, the concept of printing emojis in a program. I used assembly language for this, because I’m not entirely sure of the proper syntax for C, and I thought it would be faster to write if I did it in assembler. I tested several codes, and then when I found the result, I wrote a comment in the source file indicating what the emoji was. Be warned – this code is extremely tedious, even for an assembly program.


 1 global start
 2 
 3 segment .data
 4 str1:   db      0xf0,0x9f,0x8d,0xb0,0x0a,0x00 ; Cake emoji
 5 str2:   db      0xf0,0x9f,0x8d,0xba,0x0a,0x00 ; Beer emoji
 6 str3:   db      0xf0,0x9f,0x8d,0xa9,0x0a,0x00 ; Donut emoji
 7 str4:   db      0xf0,0x9f,0x8d,0x8e,0x0a,0x00 ; Apple emoji
 8 str5:   db      0xf0,0x9f,0x8d,0x81,0x0a,0x00 ; Maple leaf emoji
 9 str6:   db      0xf0,0x9f,0x8d,0x82,0x0a,0x00 ; Leaf emoji
10 str7:   db      0xf0,0x9f,0x8d,0x97,0x0a,0x00 ; Drumstick emoji
11 
12 segment .text
13 start:
14         push    dword 5
15         push    dword str1
16         push    dword 1
17         mov     eax,4
18         sub     esp,4
19         int     0x80
20 
21         push    dword 5
22         push    dword str2
23         push    dword 1
24         mov     eax,4
25         sub     esp,4
26         int     0x80
27 
28         push    dword 5
29         push    dword str3
30         push    dword 1
31         mov     eax,4
32         sub     esp,4
33         int     0x80
34 
35         push    dword 5
36         push    dword str4
37         push    dword 1
38         mov     eax,4
39         sub     esp,4
40         int     0x80
41 
42         push    dword 5
43         push    dword str5
44         push    dword 1
45         mov     eax,4
46         sub     esp,4
47         int     0x80
48 
49         push    dword 5
50         push    dword str6
51         push    dword 1
52         mov     eax,4
53         sub     esp,4
54         int     0x80
55 
56         push    dword 5
57         push    dword str7
58         push    dword 1
59         mov     eax,4
60         sub     esp,4
61         int     0x80
62 
63         push    dword 0
64         mov     eax,1
65         sub     esp,4
66         int     0x80

Here is what I get when I run the program:

emoji-run

It’s not particularly useful, especially considering that these emojis are not portable to other systems, but it’s an interesting concept to test.  It’s kind of confusing that the emojis use four bytes each, whereas the UTF-8 character set used by the terminal uses three bytes for each Unicode character.  I’m not sure how Apple implemented this.

Creating an MS-DOS floppy image from a directory in Unix

For a long time I have needed a way to transfer data into my DOS virtual machines. I came somewhat close to a solution by using Keka to archive directories as ISO files, but I was unable to create what I truly needed – a floppy disk image containing the archived contents of a directory. Well, now I’ve found a way to do just that, partly by doing some research on Google and partly by just figuring stuff out on my own.

The first thing you need to do is create a 1.44 MB empty file with an extension of .ima, .img, or some other raw floppy image extension. This can be done using dd, with the following command:


dd if=/dev/zero of=floppy.img bs=1024 count=1440

I took a screenshot of the output of dd for a visual:

dd

The next step is to format the file with an MS-DOS FAT filesystem.  The easiest way to do this is in DOS.  So insert the blank disk image in the VM and type format a:

format

Now you have a blank floppy image and are ready to add files to it.  To verify that the floppy had been formatted, I ran a hexdump in the Unix terminal.

hexdump

Next you need to mount the floppy image.  The easiest way to do this is to just double click on its icon in the GUI, then copy and paste the files from the source directory or directories to the directory that the image is mounted on.  I tested this first with a couple of simple standalone programs – Visicalc and DOS Cal:

adrive

Now I have installed two programs: CAL.EXE and VC.COM. Indeed, when I go to the C: drive and type cal, the program starts.

cal

Just a side note here: the default path in MS-DOS is set to C:\DOS, so if you want to run programs from a different directory you need to edit the search path in the AUTOEXEC.BAT file:

edit-path

For some reason, my DOS VM would only read the first floppy image that I created. Creating further floppy images using the same technique resulted in an Abort-Retry-Fail error message. To work around this, I have written a C program that automates the process of creating and formatting the floppy image, using the hexdump of the original image. That way I know the file will be exactly the same and I’ll be able to create multiple floppy images for software that requires a multi-disk installation. I will talk about this program, as well as MS-DOS 6.22, MS-DOS disk labels, and some additional software that I’ve installed in the next few blog entries. For now, farewell.

How to diagnose network connectivity problems from the command line

It happens to all of us. You try to visit a website, and all you get is a blank page, with a slowly spinning wheel of death in the browser tab. Why can’t you connect? The cause of your problem could lie at any of several locations between you and the website. Here, I will tell you how to find exactly where it is. (The following commands are available in both Unix/Linux and Windows/DOS, so they will probably be applicable with whatever operating system you are using.)

Let’s say you’re trying to get to wordpress.com and you encounter connectivity problems. First, you need to test your network interface card (NIC) to see if it’s working. You do this by testing the loopback interface, which is represented by the IP address 127.0.0.1.

# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.065 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.056 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.051 ms

--- 127.0.0.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.051/0.056/0.065/0.005 ms

Looks like the interface is working. If you ping an address and you get a message than says network timeout or something similar, you know you’ve encountered a dead link. Remember that there are several links, several components, that must work, and if just one of them fails, the entire connection fails.

The next step is to ping the local gateway. To do this, we can take advantage of the fact that on most home and SOHO networks, the local gateway is also the default DNS server. We find the IP address of the local DNS server with the nslookup command.

nslookup is an interactive program for interacting with DNS servers. It is a simple program with only a handful of commands. The only commands we will need here are host and exit.

# nslookup
> host
Server:		75.75.75.75
Address:	75.75.75.75#53

> exit

Now we ping the local DNS server to see if we can connect to it. If we can’t, one of two things is happening: either the default gateway isn’t working properly, or the connection between our computer and the default gateway is not working.

# ping 75.75.75.75
PING 75.75.75.75 (75.75.75.75): 56 data bytes
64 bytes from 75.75.75.75: icmp_seq=0 ttl=58 time=25.159 ms
64 bytes from 75.75.75.75: icmp_seq=1 ttl=58 time=29.576 ms
64 bytes from 75.75.75.75: icmp_seq=2 ttl=58 time=20.783 ms
64 bytes from 75.75.75.75: icmp_seq=3 ttl=58 time=20.518 ms
--- 75.75.75.75 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 20.518/24.009/29.576/3.705 ms

So that’s working. Now what we have to do is see if the default gateway can connect to the Internet. We do this by pinging one or more websites that are not the website we tried to visit, like say, google.com, yahoo.com, amazon.com, etc. If these are working, we proceed to the next step.

The next step is to ping the host we are trying to reach. If this is not working, we know that there’s a problem with that server.

# ping wordpress.com

Let’s say that works. Now we proceed to the final step. If all of these links work, then we know it’s probably a problem with the particular port we are trying to access. Let’s say we are trying to go through HTTP. The port number for HTTP is 80. To test a port we need a different tool – the Telnet protocol. Telnet can be used to access any port on a remote host. In this case we are using it to test that port. The command is as follows:

# telnet wordpress.com 80
wordpress.com:80: nodename nor servname provided, or not known

Okay, maybe it wouldn’t look exactly like that, but I couldn’t completely replicate the problem to get the proper error message.

Anyway, now we have found our problem. Port 80 isn’t working. We can either wait for it to start working again, or write to the owners of the server to inform them of the problem.

An introduction to Unix archiving with tar and cpio

One of the principles of backup and recovery is that you should back up your files in as many formats as possible; that way if one format is discontinued, or the new version is incompatible with the old version, you don’t lose the ability to recover files from your backups. In this post I will share my knowledge of two Unix archiving utilities: tar and cpio. Both of these are non-interactive programs that can be run from the command line, and many more complex backup utilities (including graphical ones) are actually frontends for these and other command line utilities.

First of all, what is an archive? An archive is the result of taking a directory or a hierarchy of directories and merging it into a single file. That’s all it is (aside from the headers and footers of course). If you do a dump of an archive file with a program like less, you will see the contents of your files concatenated together. An archive is just a bunch of files mushed together into one huge file – no compression or encryption involved. Archiving is useful when you want to compress a directory or directory tree, or when you want to encrypt a bunch of files all at the same time (but these steps are of course separate from archiving).

Part 1: tar:

First I will go over tar. tar archives files in the TAR format, which stands for Tape ARchive. TAR files are typically compressed using the GNU Zip utility so they become .tar.gz or .tgz files. This format is used for distributing software in source form. It is also used as the main package format for some Linux distributions, including Slackware.

You create an archive with the -c option. Optionally, you can add -v for verbose output.

bash-3.2$ tar -cv Screenshots > Screenshots.tar
a Screenshots
a Screenshots/Arch Linux Startup.png
a Screenshots/Arch+Linux+top.png
a Screenshots/Arch+Setup+CLI.png
a Screenshots/Arch+Setup+MDI.png
a Screenshots/Arch-Linux-top.png
a Screenshots/Arch-mc.png
a Screenshots/Arch-top.png
a Screenshots/Arch_Linux_top.png
a Screenshots/crontab.png
a Screenshots/crontab~.png
a Screenshots/Cyberdogs-Level2.png
a Screenshots/Device_manager.png
a Screenshots/Elinks+Arch+Linux.png
a Screenshots/graphics-driver.png
a Screenshots/Installing ReactOS 1.png
a Screenshots/Installing ReactOS 10.png
a Screenshots/Installing ReactOS 11.png
a Screenshots/Installing ReactOS 12.png
a Screenshots/Installing ReactOS 13.png
a Screenshots/Installing ReactOS 14.png
a Screenshots/Installing ReactOS 2.png
a Screenshots/Installing ReactOS 3.png
a Screenshots/Installing ReactOS 4.png
a Screenshots/Installing ReactOS 5.png
a Screenshots/Installing ReactOS 6.png
a Screenshots/Installing ReactOS 7.png
a Screenshots/Installing ReactOS 8.png
a Screenshots/Installing ReactOS 9.png
a Screenshots/irix-3.3-img2.gif
a Screenshots/Log_file_troubleshooting_Slackware.png
a Screenshots/Lynx.png
a Screenshots/mc-menu.png
a Screenshots/mc-mono.png
a Screenshots/memtest.png
a Screenshots/Notepad.png
a Screenshots/pkgtool-1.png
a Screenshots/pkgtool-2.png
a Screenshots/ReactOS 1.png
a Screenshots/ReactOS 2.png
a Screenshots/ReactOS 3.png
a Screenshots/ReactOS 4.png
a Screenshots/ReactOS 5.png
a Screenshots/ReactOS_Command_prompt.png
a Screenshots/ReactOS_grey.png
a Screenshots/sc.png
a Screenshots/screen.png
a Screenshots/Screensaver.png
a Screenshots/serial.png
a Screenshots/solitaire.png
a Screenshots/Spash_screen.png
a Screenshots/Task_manager_1.png
a Screenshots/Task_manager_2.png
a Screenshots/Wheat_theme.png
a Screenshots/Wordpad-bug.png
a Screenshots/WordPad.png

Alternatively, you could type tar -cvf Screenshots.tar Screenshots for the same result.

Afterward, this file can be zipped using gzip.

Files are extracted from an archive with the -x option.

bash-3.2$ tar -xvf Screenshots.tar -C .
x Screenshots/
x Screenshots/Arch Linux Startup.png
x Screenshots/Arch+Linux+top.png
x Screenshots/Arch+Setup+CLI.png
x Screenshots/Arch+Setup+MDI.png
x Screenshots/Arch-Linux-top.png
x Screenshots/Arch-mc.png
x Screenshots/Arch-top.png
x Screenshots/Arch_Linux_top.png
x Screenshots/crontab.png
x Screenshots/crontab~.png
x Screenshots/._Cyberdogs-Level2.png
x Screenshots/Cyberdogs-Level2.png
x Screenshots/Device_manager.png
x Screenshots/Elinks+Arch+Linux.png
x Screenshots/graphics-driver.png
x Screenshots/Installing ReactOS 1.png
x Screenshots/Installing ReactOS 10.png
x Screenshots/Installing ReactOS 11.png
x Screenshots/Installing ReactOS 12.png
x Screenshots/Installing ReactOS 13.png
x Screenshots/Installing ReactOS 14.png
x Screenshots/Installing ReactOS 2.png
x Screenshots/Installing ReactOS 3.png
x Screenshots/Installing ReactOS 4.png
x Screenshots/Installing ReactOS 5.png
x Screenshots/Installing ReactOS 6.png
x Screenshots/Installing ReactOS 7.png
x Screenshots/Installing ReactOS 8.png
x Screenshots/Installing ReactOS 9.png
x Screenshots/._irix-3.3-img2.gif
x Screenshots/irix-3.3-img2.gif
x Screenshots/Log_file_troubleshooting_Slackware.png
x Screenshots/Lynx.png
x Screenshots/mc-menu.png
x Screenshots/mc-mono.png
x Screenshots/memtest.png
x Screenshots/Notepad.png
x Screenshots/pkgtool-1.png
x Screenshots/pkgtool-2.png
x Screenshots/ReactOS 1.png
x Screenshots/ReactOS 2.png
x Screenshots/ReactOS 3.png
x Screenshots/ReactOS 4.png
x Screenshots/ReactOS 5.png
x Screenshots/ReactOS_Command_prompt.png
x Screenshots/ReactOS_grey.png
x Screenshots/sc.png
x Screenshots/screen.png
x Screenshots/Screensaver.png
x Screenshots/serial.png
x Screenshots/solitaire.png
x Screenshots/Spash_screen.png
x Screenshots/._Task_manager_1.png
x Screenshots/Task_manager_1.png
x Screenshots/Task_manager_2.png
x Screenshots/Wheat_theme.png
x Screenshots/Wordpad-bug.png
x Screenshots/WordPad.png

This will create a directory called Screenshots in the current directory containing all the files archived in the TAR file.

Another option is to list a table of contents for the archive, without extracting it. This is done as follows:

bash-3.2$ tar -tf Screenshots.tar
Screenshots/
Screenshots/Arch Linux Startup.png
Screenshots/Arch+Linux+top.png
Screenshots/Arch+Setup+CLI.png
Screenshots/Arch+Setup+MDI.png
Screenshots/Arch-Linux-top.png
Screenshots/Arch-mc.png
Screenshots/Arch-top.png
Screenshots/Arch_Linux_top.png
Screenshots/crontab.png
Screenshots/crontab~.png
Screenshots/._Cyberdogs-Level2.png
Screenshots/Cyberdogs-Level2.png
Screenshots/Device_manager.png
Screenshots/Elinks+Arch+Linux.png
Screenshots/graphics-driver.png
Screenshots/Installing ReactOS 1.png
Screenshots/Installing ReactOS 10.png
Screenshots/Installing ReactOS 11.png
Screenshots/Installing ReactOS 12.png
Screenshots/Installing ReactOS 13.png
Screenshots/Installing ReactOS 14.png
Screenshots/Installing ReactOS 2.png
Screenshots/Installing ReactOS 3.png
Screenshots/Installing ReactOS 4.png
Screenshots/Installing ReactOS 5.png
Screenshots/Installing ReactOS 6.png
Screenshots/Installing ReactOS 7.png
Screenshots/Installing ReactOS 8.png
Screenshots/Installing ReactOS 9.png
Screenshots/._irix-3.3-img2.gif
Screenshots/irix-3.3-img2.gif
Screenshots/Log_file_troubleshooting_Slackware.png
Screenshots/Lynx.png
Screenshots/mc-menu.png
Screenshots/mc-mono.png
Screenshots/memtest.png
Screenshots/Notepad.png
Screenshots/pkgtool-1.png
Screenshots/pkgtool-2.png
Screenshots/ReactOS 1.png
Screenshots/ReactOS 2.png
Screenshots/ReactOS 3.png
Screenshots/ReactOS 4.png
Screenshots/ReactOS 5.png
Screenshots/ReactOS_Command_prompt.png
Screenshots/ReactOS_grey.png
Screenshots/sc.png
Screenshots/screen.png
Screenshots/Screensaver.png
Screenshots/serial.png
Screenshots/solitaire.png
Screenshots/Spash_screen.png
Screenshots/._Task_manager_1.png
Screenshots/Task_manager_1.png
Screenshots/Task_manager_2.png
Screenshots/Wheat_theme.png
Screenshots/Wordpad-bug.png
Screenshots/WordPad.png

Part 2: cpio:

cpio is different from tar. Unlike tar, it creates archives in the PAX format. PAX stands for Portable Archive eXchange. Also unlike tar, cpio reads the file list from standard input and writes to standard output.

Here is a typical cpio command for archiving a directory:

bash-3.2$ ls | cpio -oacvB > Screenshots.pax
Arch Linux Startup.png
Arch+Linux+top.png
Arch+Setup+CLI.png
Arch+Setup+MDI.png
Arch-Linux-top.png
Arch-mc.png
Arch-top.png
Arch_Linux_top.png
Cyberdogs-Level2.png
Device_manager.png
Elinks+Arch+Linux.png
Installing ReactOS 1.png
Installing ReactOS 10.png
Installing ReactOS 11.png
Installing ReactOS 12.png
Installing ReactOS 13.png
Installing ReactOS 14.png
Installing ReactOS 2.png
Installing ReactOS 3.png
Installing ReactOS 4.png
Installing ReactOS 5.png
Installing ReactOS 6.png
Installing ReactOS 7.png
Installing ReactOS 8.png
Installing ReactOS 9.png
Log_file_troubleshooting_Slackware.png
Lynx.png
Notepad.png
ReactOS 1.png
ReactOS 2.png
ReactOS 3.png
ReactOS 4.png
ReactOS 5.png
ReactOS_Command_prompt.png
ReactOS_grey.png
Screensaver.png
Screenshots.paxcpio: Screenshots.pax: Can't add archive to itself

Spash_screen.png
Task_manager_1.png
Task_manager_2.png
Wheat_theme.png
WordPad.png
Wordpad-bug.png
crontab.png
crontab~.png
graphics-driver.png
irix-3.3-img2.gif
mc-menu.png
mc-mono.png
memtest.png
pkgtool-1.png
pkgtool-2.png
sc.png
screen.png
serial.png
solitaire.png
3484 blocks

Both the command structure and the output look different from those of tar. Here, the output of ls is piped into the cpio program, which then has its output redirected to the file Screenshots.pax. The verbose output shows not just a list of files, but also the number of blocks transferred.

There are three basic options for cpio: -o tells the program to produce an archive file as output; -i tells it to take an archive file as input; and -p tells it to read a list of files from standard input and copy them to a specified directory.

cpio has a very nifty feature – it gives you the option to not change the atime (access time) values of the files you are archiving. This is accomplished with the -a switch. cpio does this by saving the old access times of each of the files, then resetting them to those values when it is done archiving the directory.

The -c option tells cpio to use the ASCII header format. This makes the archive more portable.

-v is of course the switch for verbose output. Without this, the program basically operates silently, with no indication of what it’s currently doing.

-B tells cpio to use blocks that are ten times the size of the blocks used by the operating system. This can make archiving and unarchiving more efficient. If you want to specify another size for the blocks, you can use the -C switch, followed by the number of bytes in each block.

Another difference between cpio and tar is that cpio uses the same basic switch for both extracting an archive and displaying a table of contents.

Here is a cpio command for extracting an archive:

bash-3.2$ cat Screenshots.pax | cpio -iv

I have omitted the verbose output here, because it looks pretty much the same as for the other cpio command. Here I have piped the contents of the archive into cpio from the cat command, and I’ve used the -i switch to tell it to take the archive file as input. There is no need to redirect the output of cpio because it essentially has no output.

Finally, here is the command for displaying the contents without extracting:

bash-3.2$ cat Screenshots.pax | cpio -it

That’s all for now.

NOTE: Feedback and corrections to this tutorial are welcome. I will be happy to correct any mistakes people point out. Also, if anyone can tell me how to get rid of those little black rectangles above and below the code blocks, that would be great. EDIT: Fixed