David Laube's Blog » Hosting

mysql Innodb – Table ‘user’ is marked as crashed and should be repaired

admin — Sat, 08 Jan 2011 16:42:41 +0000

I recently came across a Cpanel server (CentOS 5) upon which mysqld refused to start after /var was at 100%. After tailing the mysql error log in the default /var/lib/mysql/HOSTNAME.err it was no surprise to find that the mysql user table had been marked as crashed. On RHEL/CentOS servers, you cannot simply add the “innodb_force_recovery = 1″ (or whatever recovery level…2,3,4,5,6) to the /etc/my.cnf and do the regular service mysql start. You’ll have to edit the my.cnf to enable recovery and start mysql from the command line and not by the init script/service command. Only once you have mysql started on the command line can you run your repair on the mysql user table. Here is quick run-down with commands and queries to run on your Cpanel server;

[root@HOSTNAME mysql]# tail -f /var/lib/mysql/HOSTNAME.err
110108 10:37:45  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 1 3749263016.
InnoDB: Doing recovery: scanned up to log sequence number 1 3749263050
InnoDB: Last MySQL binlog file position 0 79, file name ./host89-bin.000005
110108 10:37:45  InnoDB: Flushing modified pages from the buffer pool...
110108 10:37:45  InnoDB: Started; log sequence number 1 3749263050
InnoDB: !!! innodb_force_recovery is set to 1 !!!
110108 10:37:45 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'user' is marked as crashed and should be repaired
110108 10:37:45  mysqld ended

Now, go ahead and enable innodb forced recovery by opening your /etc/my.cnf in your favorite text editor and make sure you have something like this:

[mysqld]
innodb_force_recovery = 1

Start mysql from the command line after enabling innodb forced recovery:

[root@HOSTNAME ~]#/usr/sbin/mysqld --skip-grant-tables --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/HOSTNAME.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock

Finally, it’s time to get back in to mysql and get your life back… or mysql user table at least!

[root@HOSTNAME ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 173 to server version: 4.1.22-standard-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use mysql;
mysql> check table user;
+------------+-------+----------+----------------------------------------------------------+
| Table      | Op    | Msg_type | Msg_text                                                 |
+------------+-------+----------+----------------------------------------------------------+
| mysql.user | check | warning  | Table is marked as crashed                               |
| mysql.user | check | warning  | 6 clients are using or haven't closed the table properly |
| mysql.user | check | error    | Record at pos: 24992 is not remove-marked                |
| mysql.user | check | error    | record delete-link-chain corrupted                       |
| mysql.user | check | error    | Corrupt                                                  |
+------------+-------+----------+----------------------------------------------------------+
5 rows in set (0.02 sec)

mysql> repair table user;
+------------+--------+----------+------------------------------------------+
| Table      | Op     | Msg_type | Msg_text                                 |
+------------+--------+----------+------------------------------------------+
| mysql.user | repair | warning  | Number of rows changed from 1384 to 1385 |
| mysql.user | repair | status   | OK                                       |
+------------+--------+----------+------------------------------------------+
2 rows in set (0.48 sec)

mysql> check table user;
+------------+-------+----------+----------+
| Table      | Op    | Msg_type | Msg_text |
+------------+-------+----------+----------+
| mysql.user | check | status   | OK       |
+------------+-------+----------+----------+
1 row in set (0.01 sec)

mysql>exit

Now, don’t forget to REMOVE the innodb_force_recovery line from your my.cnf you added earlier! After that, just start mysql as you normally would. For more database information and management hop over to here.

[root@HOSTNAME mysql]# service mysql start

Cpanel – incorrect email disk space used

admin — Fri, 20 Aug 2010 15:28:02 +0000

Control Panel -> Email -> Add/Remove/Manage Accounts -> Show Disk Space Used

When there are inconsistencies with Cpanel’s email disk space usage, you should verify the actual disk space being utilized by:

(1)

du -sh /home/username/mail/userdomain.com/user/
tail -1 /home/username/mail/domain.com/user/maildirsize

(2) Compare the utilized disk space reported by the first du command with the bytes shown in the user’s maildirsize file. If the totals reported are different delete or rename the maildirsize file (/home/username/mail/domain.com/user/maildirsize). Next, logout and then back in again within the user’s control panel and go to Control Panel -> Email -> Add/Remove/Manage Accounts -> Show Disk Space Used

How to determine if a SATA drive is failing.

admin — Mon, 12 Apr 2010 15:42:32 +0000

When is it a good time to check to see if a hard drive is failing? Well, when your console is full of IO/seek errors, I’d say that is a pretty good time! Hah.

According to research conducted by Google, a document entitled Failure Trends in a Large Disk Drive Population states that the manufacturer, particular model and vintage plays a role, but does not provide failure statistics on model and manufacturers. Most drives were run at 45C or less.

From the SMART data, scan errors, reallocations, offline reallocations and probational counts had a significant correlation with failure probability, whereas seek errors, calibration retries and spin retries had little significance.

Soooo…. you want to look at the Raw_Read_Error_Rate, Seek_Error_Rate and Reallocated_Sector_Ct information from smartctl.

[root@SOMESERVER ~]# smartctl --all /dev/sdb | grep Error
Error logging capability:        (0x01)	Error logging supported.
  1 Raw_Read_Error_Rate     0x000f   117   100   006    Pre-fail  Always       -       166491825
  7 Seek_Error_Rate         0x000f   090   060   030    Pre-fail  Always       -       999290467
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
SMART Error Log Version: 1

[root@SOMESERVER ~]# smartctl --all /dev/sdb| grep Reallocated_Sector_Ct5
Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0

In regards to Reallocated_Sector_Ct, the normalized values (current=100, worst=100) indicate the drive is in perfect condition (higher is better, and looking at the overall report it appears that 100 is “best”). The threshold value (36) just indicates how low the normalized value would have to drop before the manufacturer would consider the drive to be in a “Pre-fail” condition.

If you run “smartctl –all /dev/sdb | grep Error” again and notice that Raw_Read_Error_Rate and Seek_Error_Rate keep incrementing AND Reallocated_Sector_Ct is greater than 0, its pretty safe to say that you have a ticking time-bomb on your hands. You should consider replacing those drives as soon as possible.

Unknown linux rootkit?

admin — Sun, 04 Apr 2010 19:13:17 +0000

Recently noticed a bunch of servers (CentOS 5.2 and CentOS 4.8) with SSH fingerprint mismatches. After poking around, it appears that they had been compromised. chkrootkit and RKhunter found nothing but the suspicious entry in /dev (see below).

WARNING: DSA key found for host ourserver.domain.com
in /Users/username/.ssh/known_hosts:578
DSA key fingerprint f4:27:d0:32:6b:4c:9f:6e:52:6f:49:dd:19:54:c2:f1.
+–[ DSA 1024]—-+
| ..o+|
| . o..|
| * . .E|
| + B . . .o|
| S * o …|
| . o = . |
| . o + |
| o . |
| |
+—————–+

The authenticity of host ‘ourserver.domain.com (10.0.0.2)’ can’t be established
but keys of different type are already known for this host.
RSA key fingerprint is 4c:5f:95:3e:52:08:6c:cd:0f:f8:37:38:3c:dd:bf:56.
Are you sure you want to continue connecting (yes/no)? yes

Affected files:

/dev/sax
/bin/suidshell
/usr/local/bin/ssh-agent
/usr/local/bin/sftp-server
/usr/local/bin/ssh-agent2
/usr/local/bin/ssh
/usr/local/bin/ssh-add2
/usr/local/bin/ssh-signer
/usr/local/bin/ssh-keygen2
/usr/local/bin/ssh-probe2
/usr/local/bin/ssh-probe
/usr/local/bin/scp
/usr/local/bin/sftp
/usr/local/bin/ssh-chrootmgr
/usr/local/bin/ssh-signer2
/usr/local/bin/sftp2
/usr/local/bin/scp2
/usr/local/bin/ssh-pubkeymgr
/usr/local/bin/ssh-dummy-shell
/usr/local/bin/ssh-add
/usr/local/bin/sftp-server2
/usr/local/bin/ssh-askpass
/usr/local/bin/ssh2
/usr/local/bin/ssh-keygen
/usr/local/sbin/sshd2
/usr/local/sbin/sshd-check-conf
/usr/local/sbin/sshd

/usr/local/sbin/sshd2

/bin/suidshell — is exactly that, a shell with the suid bit set that instantly gives root access to any unprivileged user!

/dev/sax — stores plaintext username and password for root and non-root accounts

eAccelerator with suphp woes

admin — Mon, 29 Mar 2010 15:46:25 +0000

It would appear that both mmcache and eaccelerator complain when setup under a suphp implementation for PHP and apache. Seems like most people recommend using fastcgi+suexec and then a php accelerator, but I’m uncertain how that will turn out as of right now. Another option on a per user basis would be to use Alternative PHP Cache (APC )

pure-ftpd Can’t change directory to /var/ftp/

admin — Wed, 24 Mar 2010 15:17:33 +0000

The problem arises when a user attempts to make an anonymous FTP connection to Cpanel user’s account who has already enabled anonymous FTP connections in their control panel. However, pure-ftpd drops you with the error “421 Can’t change directory to /var/ftp/”.

workstation:~ user$ ftp testing.com
Connected to testing.com.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 3 of 50 allowed.
220-Local time is now 11:02. Server port: 21.
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (testing.com:user): anonymous
421 Can't change directory to /var/ftp/ [/]
ftp: Login failed.

The solution(s):
1. Use anonymous@domain.com and any password instead of just anonymous
…OR
2. Assign the Cpanel user a dedicated IP address where FTP logins with just “anonymous” will work.

apache_conf_distiller User data set has no ‘main_domain’ key.

admin — Tue, 23 Feb 2010 01:55:41 +0000

After a server was hacked recently, the attackers replaced all files named index / default / main with their typical 0wned-by message and javascript. Of the “main” files that were affected, the /var/cpanel/userdata/USER/main files were also overwritten. These files contain important cpanel domain information which are required to build a new httpd.conf using the apache_conf_distiller. The following steps should regenerate a working apache userdata in order to fix subdomains. Thanks to Josh for finding userdata_update!

/etc/init.d/httpd stop
mv /usr/local/apache/conf/httpd.conf /usr/local/apache/conf/httpd.conf-notworking
cp -a OLDHTTPD.CONF /usr/local/apache/conf/
mv /var/cpanel/userdata /var/cpanel/userdata-BAK
/usr/local/cpanel/bin/userdata_update
cp -a /var/cpanel/userdata /usr/local/apache/conf
/etc/init.d/httpd start

Run the /usr/local/cpanel/bin/apache_conf_distiller –update to ensure the main_domain key errors are gone.