Linux Lifechangers

Today I want to write about the discoveries that have made my Linux life easier. What are your discoveries? Add them in the comment section. Here are mine:

ClusterSSH

to execute commands on several computers simultaneously (http://www.linuxintro.org/wiki/Clusterssh):

ClusterSSH: type a command once - have it run on two computers

shellinabox

to bring your Linux shell into the browser (http://www.linuxintro.org/wiki/Shell_in_a_box)

guacamole

to bring your Linux desktop into the browser (http://www.linuxintro.org/wiki/Guacamole). No browser plugin needed!

it's a remote Linux desktop - and it's in Firefox

scp

to copy files to or from a remote computer over the network. It is possible whereever ssh connectivity is possible, so, mostly between two Linux computers. Its use is quite simple:
scp file user@remotehost:/folder
and
scp user@remothost:/folder file

Webex

to share a desktop. Give your trainees a training over the web, show them your desktop and how you do your work. Get support, show some support engineers where you struggle. Also, get training and give support. Record what you do on your desktop in a video. I describe how I get it running under Linux here: http://www.linuxintro.org/wiki/Webex.

Rackspace

to get a Linux server provisioned in less than 5 minutes. Your fee is calculated by the minute. Similar: Amazon Web Services.

Send a program to the background

How often have you started a program in a shell and then wanted to start the next program with the current program still running? The solution is to send the program to the background:

• stop it with the key combination CTRL_Z
• resume it in the background with the command

bg

wiki

to publish content on the web and be able to edit it with a browser. And to track your changes: http://www.linuxintro.org/wiki/Wiki

strace

to find out what a program does and to find out where is passes most of its time. I blogged about it.

See also

• http://www.linuxintro.org/wiki/Tips_and_tricks
• https://www.linkedin.com/groups/what-tools-made-your-life-65688.S.5889720806715437057

how to strace a process

strace is a Linux command that can help you with error analysis. By default it will output every syscall a process does. This means

• it will output a lot of lines that you will have to examine
• in the worst case your problem will not even show up. Consider an endless loop like while (true); - this one does not do any syscalls at all.

There is no tool that "drills down" deeper on this analysis than strace. By this it is comparable to

• disassembling a program as I give an example here. A program, as a contrast to a process, is something stored on disk rather than running in RAM. Disassembling will reveal problems like the above endless loop, but it does not work on a running process. 
• network sniffing like I give an example here. There is no drill-down to looking at the bytes that are transmitted via a network cable
• sniffing the SCSI bus with blktrace like I give an example here
• sniffing the USB bus with wireshark like I give an example here
• testing IMAP via telnet - going down to the very elements defined in the IMAP protocol

It takes some time till you can use strace for an actual purpose. I started with Linux in 2000 and the first time strace helped me to accomplish a task was in 2013. Here is what happened:

I was setting up guacaMole, the tool to control a Linux desktop from a browser. It basically consists of a daemon, guacd that (if I understood it right) accepts data from the browser and uses it to control a VNC session. It gave me the error message "Server error" which does not really mean anything. So I strace'd it:

# ps -A|grep guac

 7361 ?        00:00:00 guacd

# strace -p 7361

for every login I did on the web interface I saw this in strace's output:

accept(4, {sa_family=AF_INET, sin_port=htons(42845), ...

clone(child_stack=0, ...

close(5)                                = 0

listen(4, 5)                            = 0

now the commands "man 2 accept", "man 2 clone", "man 2 close" and "man 2 listen" told me more. "Clone" means that a separate process will be spawned off. We want strace to follow this process as well so we call it again with the ff parameter:

# strace -ffp 7361

This time there was more output. There were several lines like

[pid 20344] open("/usr/lib/x86_64-linux-gnu/libguac-client-vnc.so", O_RDONLY) = -1 ENOENT (No such file or directory)

reading the complete output, this libguac-client-vnc.so was never found. This told me I had to re-compile the guacamole server with VNC development libraries so its VNC module would be buildable. When I did this, I also had to move the library to the right place (make install did not do), but these are the little hurdles described in my guacaMole tutorial.

You can do much more with strace. If I had known this was all about a missing file I would have started it with -e open. This tells strace to only print occurences of the syscall open. In effect it allows you to monitor which files are accessed by a process. I use it to find out where configuration changes are stored (ok, maybe this was not the first time strace was useful to me).

You do not need to know a process ID to call strace. You can just call 

strace program

and strace will load program as a process and output its syscalls.

strace shortens information for you and replaces strings by "..." before they get too long. To change this use the -s parameter to determine how long strings are supposed to be. For more information see the man page.

strace can also be used a bit more abstract for performance analysis.

strace -c ls -R

will show you in which syscalls the ls program passed most of its time. This is helpful if you want to optimize routines for speed. The output could look like this:

Entries  Repository  Root

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

 67.09    0.412153          14     29664           getdents64

 27.70    0.170168          11     14849        14 open

  4.24    0.026043           0    123740           write

  0.72    0.004443           0     14837           close

  0.20    0.001204           0     14836           fstat

  0.05    0.000285         285         1           execve

  0.00    0.000000           0        12           read

  0.00    0.000000           0         4         3 stat

  0.00    0.000000           0        33           mmap

  0.00    0.000000           0        18           mprotect

  0.00    0.000000           0         4           munmap

  0.00    0.000000           0        12           brk

  0.00    0.000000           0         2           rt_sigaction

  0.00    0.000000           0         1           rt_sigprocmask

  0.00    0.000000           0         2           ioctl

  0.00    0.000000           0         1         1 access

  0.00    0.000000           0         3           mremap

  0.00    0.000000           0         1           fcntl

  0.00    0.000000           0         1           getrlimit

  0.00    0.000000           0         1           statfs

  0.00    0.000000           0         1           arch_prctl

  0.00    0.000000           0         3         1 futex

  0.00    0.000000           0         1           set_tid_address

  0.00    0.000000           0         8           fadvise64

  0.00    0.000000           0         1           set_robust_list

------ ----------- ----------- --------- --------- ----------------

100.00    0.614296                198036        19 total

which means that more than two thirds of ls' time was passed in the getdents64 syscall. I'll leave you alone for now if you want to man 2 getdents64 and start optimizing the ls program to list directories. Have fun!

read e-books with Linux

Yes! I read an e-book on my Linux computer \o/

I did this on 2013-10-25 with SUSE Linux 12.2:

• go to http://www.amazon.com and get some free e-books.
• downloaded their Kindle reader for Windows
• delete the old version of the Windows Emulator wine from my computer. To do this, open a console and type

    rpm -e wine wine-32bit wine-mp3

• I knew that I have a 64bit computer. If you do not know, find out if your system is 64bit or 32bit
• download wine 1.7.4 and wine-32bit 1.7.4 for SUSE Linux from http://download.opensuse.org/repositories/Emulators:/Wine/openSUSE_12.2/x86_64/
• and wine-gecko from http://download.opensuse.org/repositories/Emulators:/Wine/openSUSE_12.2/noarch/
• Then install it:

    rpm -ivh wine-1.7.4-*.rpm wine-gecko-*.rpm wine-32bit-*.rpm

• Now you start the reader's installation:

    wine Downloads/KindleForPC-installer.exe

• And start the reader:

    cd .wine/drive_c/Program\ Files/Amazon/Kindle
    wine Kindle.exe

pretty URL in mediawiki

For my personal homepage http://www.staerk.de/thorsten I use mediawiki. That way I can edit it whenever I want, even from an internet cafe or the like.

When I write about e.g. my Cybermaster I do not want it to appear anywhere but on http://www.staerk.de/thorsten/cybermaster. This is called a prettyURL and I want to show here how I do it. My URL http://www.staerk.de/thorsten corresponds to the directory /var/www/staerk.de/thorsten on the server. I am using mediawiki 1.21.2 on Apache2 under Ubuntu Linux 10.04

• make sure Apache has the "rewrite" module loaded:

    # apache2ctl -t -D DUMP_MODULES 2>&1 | grep rewrite

    rewrite_module (shared)

• configure apache to rewrite your requests as discussed here:

    RewriteRule ^/?thorsten(/.*)?$ %{DOCUMENT_ROOT}/thorsten/index.php [L]

    RewriteRule ^/*$ %{DOCUMENT_ROOT}/thorsten/index.php [L]

• restart apache

    /etc/init.d/apache2 restart

• Add the following line to LocalSettings.php:

    $wgArticlePath = "/thorsten/$1";

Now your wiki will look "strange" like this:

The reason for this "strange" look is that the skins are missing. Mediawiki is using a file called load.php and when loading it, apache will use its rewrite-rules and try to load it as a wiki page. The solution is:

• add the following condition above the first rewrite rule:

    RewriteCond %{REQUEST_URI} !^/thorsten/load.*$

Then it looks better, but still the images are missing. No problem, reading apache's access log at /var/log/apache2/other_vhosts_access.log I find that clients try to download them from /thorsten/Images. So

• add another condition above the rewrite rule:

    RewriteCond %{REQUEST_URI} !^/thorsten/images.*$

and the page looks again as it should:

See also

• http://www.staerk.de/thorsten/Updating_mediawiki_to_1.19.1

Error creating thumbnail: Unable to save thumbnail to destination

I just set up mediawiki on http://www.staerk.de/thorsten. Whenever I upload an image I see the error message:

Error creating thumbnail: Unable to save thumbnail to destination

1. Permissions

Now the first thing you should check when you get this error message is that your server's directories are writeable to the web server process. This is Ubuntu Linux so as long as the files are owned by www-data everything is okay. They were so this is not the problem:

/var/www/staerk.de/thorsten# ll

-rw-r--r--  1 www-data www-data     1088 Sep  3 18:56 thumb_handler.php

-rw-r--r--  1 www-data www-data      956 Sep  3 18:56 thumb_handler.php5

-rw-r--r--  1 www-data www-data       86 Sep  3 18:56 wiki.phtml

[...]

2. Logs

So I had to dig deeper. I looked into the server log if there was any entry at the time when I uploaded:

cat /var/log/apache/error_log

There was no message at these times so I moved on.

3. PHP's safe_mode

php's safe_mode feature is another thing that can make accessing files impossible. safe_mode is a great feature of PHP and it should never be turned off except for experimenting and troubleshooting. So I opened /etc/php5/apache2/php.ini, searched for safe_mode and set it to off:

safe_mode = Off

restarted the web server with the command

/etc/init.d/apache2 restart

and during the next time when I uploaded an image the error was gone.
So I set safe_mode to On again, restarted apache and entered the following line into /var/www/staerk.de/thorsten/LocalSettings.php:
putenv('TMPDIR=/var/www/staerk.de/thorsten');
Now whenever I upload a file I get a warning in /var/log/apache2/error.log like:

[Fri Oct 04 08:37:50 2013] [error] [client 95.113.203.63] PHP Warning:  putenv(): Safe Mode warning: Cannot set environment variable 'TMPDIR' - it's not in the allowed list in /var/www/staerk.de/thorsten/LocalSettings.php on line 31, referer: http://www.staerk.de/thorsten/index.php/Special:Upload

So what is in the allowed list?
Looking into /etc/php5/apache2/php.ini helps. I set it to empty which allows to change all environment variables:
safe_mode_allowed_env_vars =
And the error changes. The error now says:
Error creating thumbnail: Unable to run external programs in safe mode.
Googling helped. In /var/www/staerk.de/thorsten/LocalSettings.php I added at the end:
$wgUseImageMagick = false
and it works as you can see on http://www.staerk.de/thorsten

my virtual machine is getting slow

I have been asked multiple times "why is my virtual machine getting slower after some time". I have heard this about VMware and VirtualBox machines without a lot of concrete information what happens.

It happens to me as well. It seems to have something to do with (a) time and (b) snapshots that you have created. It causes a machine to become really slow. Here is a "benchmark" to show what I mean. It writes zeroes to memory as fast as possible. On my VirtualBox virtual machine I get:

linux-s0l1:~ # dd if=/dev/zero of=/dev/shm/testfile bs=8M count=10

10+0 records in

10+0 records out

83886080 bytes (84 MB) copied, 2.0009 s, 41.9 MB/s

42 MB/s when writing to RAM? Wow that's bad. Let's look what the host machine makes:

tweedleburg:~ # dd if=/dev/zero of=/dev/shm/testfile bs=8M count=10
10+0 records in
10+0 records out
83886080 bytes (84 MB) copied, 0.0423951 s, 2.0 GB/s

about 50 times as much. Actually, even moving my mouse on the VM's display make the CPU go wild!

In VirtualBox, I deleted all snapshots and my virtual machine's performance went up from 42 MB/s to 150 MB/s. But I have other VirtualBox virtual machines on the same host with the same guest operating system that deliver 1.8 MB/s.

booting from network

In the lab, I can boot any server from the network. This allows me to start up any distribution I like. Also it is possible to install Linux by booting via the network as I describe in 3 ways to install Linux. The technology to  boot from the network is called PXE, preboot execution environment. Today I want to use it for my virtual machines. The setup looks like this:

On my main computer I use SUSE Linux.

DHCPD setup

A dhcp daemon assigns a dynamic IP address to a computer that connects to the network. The PXE technology requires this, so let's set up a DHCP server (daemon) on the main computer. Open a console as root and call:

yast -i dhcp-server

Make sure it listens on network device eth1. To do this, open /etc/sysconfig/dhcp, set 

DHCPD_INTERFACE=eth1

We don't use eth0. eth0 gets its IP address from the router which interferes with our goal.

Configure DHCP with a file /etc/dhcpd.conf:

subnet 192.168.0.0 netmask 255.255.255.0

{

   range 192.168.0.10 192.168.0.100;

   next-server 192.168.0.1;

   filename "pxelinux.0";

}

Start dhcpd using the command

/etc/init.d/dhcpd start

Activate dhcpd start on boot using the command

chkconfig dhcpd on

TFTP setup

TFTP, the trivial File Transfer Protocol, allows for transmitting bootcode over the network. Install the server using the command:

yast -i tftp

Then edit /etc/xinet.d/tftp, replace

disable = yes

by

disable = no

Then restart xinetd, the service hosting tftp:

/etc/init.d/xinetd restart

Now test your tftp server using the command

tftp localhost -c get pxelinux.0
There should not be any error message from this command. Then activate xinetd start on boot:
chkconfig xinetd on 

RIPLinux setup

RIP Linux is a distribution optimized for (recovery and ) PXE.

cd /srv
wget http://www.tux.org/pub/people/kent-robotti/looplinux/rip/RIPLinuX-13.7.PXE.zip
unzip RIPLinuX-13.7.PXE.zip

VirtualBox setup

VirtualBox allows you to create and run virtual machines. These virtual machines can boot from LAN (local area network). To set it up, open a console as root and call:
yast -i virtualbox
or, depending on your SUSE version
yast -i virtualbox-ose
Start it with the command
VirtualBox
Then create a virtual machine by clicking onto the "New" Button.

Now select your new virtual machine, click on "Settings" -> "Network". You want to make it a bridged adapter (in my case bridge to eth1) and select "advanced" -> "Adapter Type". As adapter type, set PCnet-Fast III, even if this is not build into your computer. If you do not select this, PXE will not work:

To boot your virtual machine from network (LAN), click on "Start", then type F12, then L.