Recovering Ubuntu 8.04 LTS from Failed Harddisk

You may have noticed that the web server hosting my homepage and download site has been down since the weekend. I noticed problems when the sites responded to request with MySQL errors.

Running fsck returned the message “Bad magic number in super-block” which means that e2fsck cannot completely repair the disk. It turned out that the partition table had been destroyed, but mke2fs -n still found some superblocks.

Started up the PC with a Knoppix Live CD to repair the broken disk.

In couple of forums I found the utility TestDisk which re-creates lost partitions, and it both recovered the boot and the swap partition. However, e2fsck still failed with the messages:

Attempt to read block from filesystem resulted in short read
Attempt to read block from filesystem resulted in short read
reading journal superblock
Attempt to read block from filesystem resulted in short read
while checking ext3 journal

It was now clear that the harddisk could not be repaired, so I got a new one and copied the original harddisk with a program called Ddrescue.

Ddrescue is a great tool (documentation), as it copies one device onto another, displaying the number of read errors and the size of the erroneous blocks. The amazing thing (if you don’t know how it works) is that the error size *reduces* after the first full scan of the source disk. The initial 2.5GB of unreadable disk finally reduced to about 15MB.

After ddrescue was finished, I ran another fsck on the new disk, this time successfully. Time to reboot.

Reboot brought a black screen, with “1234F” the only thing displayed. It turned out that that was the remainder of the TestDisk MBR which could not find a bootable partition. Need to get GRUB back.

The Knoppix disk would not help me now (disks are named /dev/hda instead of /dev/sda), but fortunately I had a Ubuntu 8.10 Desktop disk already which also offers a Live functionality.

Booted Ubuntu CD, and restored GRUB as sketched in this forum thread:

sudo grub
find /boot/grub/stage1
root (hd0,0)
setup (hd0)

Now at least GRUB was booting, but it also served me the next error message:

Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)

If tried to understand GRUB and the disk UUIDs mentioned in menu.lst, when I guessed that the problem was caused by a broken initrd.img.

I backed up the original initrd.img-2.6.24-16-server and copied the initrd.img-2.6.24-16-server.bak to the original name. And it worked!

As far as I can tell, the machine is back online and fully functioning again. But it was quite a trip 😉

Extending default validity for self-generated SSL certificates

When I installed GForge recently, I had to generate an SSL certificate as part of the installation since GForge runs on https. However, the make-ssl-cert tool does not provide a way to customize the validity of the generated certificate, which is 30 days by default.

As it turns out, this problem is known for 3 years, and it is still not fixed in Ubuntu 8.04.

The easiest workaround is to edit the make-ssl-cert script using

whereis make-ssl-cert
sudo nano [path-to/]make-ssl-cert

and replace the line

openssl req -config $TMPFILE -new -x509 -nodes -out $output
    -keyout $output

with

openssl req -config $TMPFILE -new -x509 -days 365 -nodes 
    -out $output -keyout $output

That’s it.

Installing GForge on Ubuntu 8.04 Hardy

I was looking for a program that lets me easily provide download and versioning functionality for the software that I create, and I decided to give GForge a try on my Ubuntu server. Installation turned out to be a bit messy, but finally I managed to make it work.

First, I tried the gforge package that came with Ubuntu using apt-get, but it failed to install the database component gforge-db-postgresql with the message

constraint "forum_group_forum_id_fk" for relation "forum" already exists

(see bug report). From this post I figured out that several constraint triggers with the same name were created in the SQL script, which had been removed according to the changelog.

After modifying the SQL script, installation raised another error with a malformed SQL statement containing

WHERE oldvalue=100 AND field_name='category_id'

(sorry I lost the original error message). Unable (and also unwilling) to debug unknown SQL code, I decided to give up.

Next, I tried the gforge package provided by Christian Bayle. I added the repository to my apt sources, and started installation. Unfortunately, the same SQL error in the WHERE clause caused the installation to terminate unsuccessfully.

I uninstalled everything using apt remove, and turned to my last choice:

Manual installation

From the download page, I chose gforge-4.5.19, because it seems to be the most recent stable version. The installation manual can be found on the Documentation page (User Documentation / Gforge Manual (PDF)). Even though the manual was written for 4.5.3, it is still accurate enough. Chapter 2.3 covers installation.

Since the earlier installation attempts already created a PostgreSQL database, I had to drop both the database and the user

sudo su - postgres
DROP DATABASE gforge;
DROP USER gforge;

The tcpip_socket option in postgresql.conf is not recognized anymore (as of version 8.3).

Setting up the Apache configuration was a bit of a mess, because the previous attempts had generated a gforge.conf (/etc/gforge/httpd.conf) file which contained instructions that Apache2 does not recognize. They also added references to modules that do not exist, such as auth_gforge_module.

Anyway, executing a manual installation from scratch should definitely work with fewer problems than I experienced.

Finally, I got the web application to run, but I had not enabled redirection which caused to redirect an HTTP request to HTTPS. Therefore, when I clicked on the link “Register as a site user”, which requires an SSL connection, the browser display the error message:

(site URL) has sent an incorrect or unexpected message
(error code -12263)

There were strong indications that the error was related to SSL. 😉

Fine, I thought, just activate the https redirection, enable the VirtualHost section in the apache config file, create an SSL certificate, and then we’re done. So, I locate the instructions to set up an SSL server, which rely on a script called apache2-ssl-certificate which is not installed on Ubuntu Hardy. Well, actually it is missing from Ubuntu versions for one and a half years now.

Fortunately, the work-around is in the comments:

sudo make-ssl-cert /usr/share/ssl-cert/ssleay.cnf
    /etc/ssl/private/gforge.pem

In the VirtualHost section, add (or uncomment) the lines

SSLEngine On
SSLCertificateFile /etc/ssl/private/gforge.pem

In the end, I could successfully register a new user, added the Admin privilege via PostgreSQL, and now I am the proud and happy owner of a gforge installation 😉

Let’s see what adventures await me there.