Leaderboard (728 x 90)
Showing posts with label Linux/Unix. Show all posts
Showing posts with label Linux/Unix. Show all posts

Monday, July 25, 2022

Zimbra Mysql Crash Recovery

In the event of database corruption it may be necessary to manually perform database recovery. See Bug 15797 for an example of an issue with mysql that will require database recovery. In that example, a warning message like the following appeared in the mysql error log:

InnoDB: Serious error! InnoDB is trying to free page 716
InnoDB: though it is already marked as free in the tablespace!
InnoDB: The tablespace free space info is corrupt.
InnoDB: You may need to dump your InnoDB tables and recreate the whole
InnoDB: database!

Before beginning a full database recovery, check to see if the corruption may be limited to a single mboxgroup or a single user within an mboxgroup. This type of corruption frequently lets the server run normally for extended periods of time, with crashes occurring only when an affected user attempts to access certain mailbox items. If this is the case, it may be possible to dump, drop and recover only the affected entries without disrupting the database as a whole. Please see the instructions in the Mysql Crash Recovery (alternate method) article.

Overview of Recovery Process

  1. Configure mysql to start in recovery mode
  2. Generate SQL dumps of all relevant databases
  3. Remove all existing (and possibly corrupt) databases
  4. Re-create all databases
  5. Repopulate the databases with the data from the SQL dumps
  6. Test databases and start all ZCS services

Details of Recovery Process

1. Configure mysql to start in recovery mode

  1. Edit the file /opt/zimbra/conf/my.cnf and add a line like innodb_force_recovery = 1 under the [mysqld] section (Note that it may be necessary to increase the recovery level depending on the extent of the database corruption, as shown at the end of the database dump step)
  2. Save the file and re-start mysqld
mysql.server start

2. Generate SQL dumps of all databases

  1. Load some mysql configuration into shell variables (i.e. $mysql_socket and $mysql_root_password; note that you will use these again in step 3)
  2. Make a list of the existing databases
  3. Create a directory to hold the SQL dumps
  4. Generate the SQL dumps from the database list
source ~/bin/zmshutil ; zmsetvars
mysql --batch --skip-column-names -e "show databases" | grep -e mbox -e zimbra > /tmp/mysql.db.list

Note: If you are using ZCS v8.8.x with Chat/Talk enabled then you should take Chat database dump as well

mysql --batch --skip-column-names -e "show databases" | grep -e mbox -e zimbra -e chat > /tmp/mysql.db.list
mkdir /tmp/mysql.sql 
for db in `cat /tmp/mysql.db.list`; do
     mysqldump $db -S $mysql_socket -u root --password=$mysql_root_password > /tmp/mysql.sql/$db.sql
     echo "Dumped $db"
     sleep 10
 done

Note: If you encounter any mysql errors while dumping the databases, start over by re-editing /opt/zimbra/conf/my.cnf, incrementing the value for innodb_force_recovery by one, and restarting mysqld. It is critical to update this incrementally - 1, 2, 3, and only if needed 4. 4 and above can cause DB corruption. Please see MySQL's Forcing InnoDB Recovery guide for more information.

Note: Starting 8.7 , path of mysqldump has been changed from /opt/zimbra/mysql/bin/mysqldump to /opt/zimbra/common/bin/mysqldump . Please update the command accordingly if you are doing this for a system >= ZCS 8.7.x

Note: An error of "bash: /tmp/mysql.sql/$db.sql: ambiguous redirect" probably indicates your using an apostrophe or single quote ' rather than a tick ` -- which is one the same key as the tilde ~ .

Note: Do not reboot the machine, as some Operating Systems will remove all contents in the /tmp directory during the reboot sequence, i.e. your /tmp/mysql.sql will be removed.

HINT Did the dump work or not, try grep -L "Dump completed" /tmp/mysql.sql/*.sql [those that didn't] and grep "Dump completed" /tmp/mysql.sql/*.sql [those that did].

3. Remove all existing (and possibly corrupt) databases

Note: Take a copy of /opt/zimbra/db/data before dropping the databases. This will ensure a copy of old database.

Note that we drop the zimbra database last because the mboxgroup* databases depend on it

for db in `cat /tmp/mysql.db.list |grep mbox`
do
    mysql -u root --password=$mysql_root_password -e "drop database $db"
    echo -e "Dropped $db"
done
mysql -u root --password=$mysql_root_password -e "drop database zimbra"

Remove existing InnoDB tablespace and log files

rm -rf /opt/zimbra/db/data/ib*

Note: First, use with caution - this shouldn't need to be used often. Issue came about because of some rsync issues. Can't dump db's because of 'connection' issues at this point? One could move the /opt/zimbra/db/data directory - mv /opt/zimbra/db/data /opt/zimbra/db/data-old and then make the db - mkdir /opt/zimbra/db/data w/ ownership of zimbra:zimbra . Remove the innodb_force_recovery line from /opt/zimbra/conf/my.cnf . Then recreate a default mysql db by running /opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password and then attempt this steps over again to confirm you can drop them. Also note that you may have to reset the zimbra password manually in mysql, then set it again in Zimbra with the instructions from this page: http://wiki.zimbra.com/wiki/Resetting_LDAP_%26_MySQL_Passwords

4. Re-create all databases

  1. Run mysql in non-recovery mode
    1. Remove the innodb_force_recovery line from /opt/zimbra/conf/my.cnf
    2. Save the file and restart mysqld
  2. Re-create the databases from the database list
mysql.server restart
for db in `cat /tmp/mysql.db.list`
do
    mysql -e "create database $db character set utf8"
    echo "Created $db"
done

5. Repopulate the databases with the data from the SQL dumps

Import the data from the SQL dumps. Note that we import the zimbra database first because the mboxgroup databases depend on it

mysql zimbra < /tmp/mysql.sql/zimbra.sql
for sql in /tmp/mysql.sql/mbox*
do
    mysql `basename $sql .sql` < $sql
    echo -e "Updated `basename $sql .sql` \n"
done

Note : If you are using ZCS v8.8.x with Chat/Talk enabled then you should import chat db as well.

mysql chat < /tmp/mysql.sql/chat.sql

6. Test databases and start all ZCS services

Note that this is an example query. If you know of any particular databases that were corrupt, you may want to construct other queries to verify normal access to the data.

mysql zimbra -e "select * from mailbox order by id desc limit 1"

Once you are satisfied that the databases are restored intact, start the rest of the zimbra services.

zmcontrol start

Check /opt/zimbra/log/mysql_error.log and /opt/zimbra/log/mailbox.log for database errors.


Reference: https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery

Monday, September 30, 2013

How do I disable IPv6?

Upstream employee Daniel Walsh recommends not disabling the ipv6 module, as that can cause issues with SELinux and other components, but adding the following to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
To disable in the running system:
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
or
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
If problems with X forwarding are encountered on systems with IPv6 disabled, edit /etc/ssh/sshd_config and make either of the following changes:
(1) Change the line
#AddressFamily any
to
AddressFamily inet
(inet is ipv4 only; inet6 is ipv6 only)
or
(2) Remove the hash mark (#) in front of the line
#ListenAddress 0.0.0.0

Then restart ssh.

About Network Interfaces in EL6

Each physical and virtual network device on an EL6 Linux system has an associated configuration file named ifcfg-interface in the /etc/sysconfig/network-scripts directory, where interface is the name of the interface. For example:
# cd /etc/sysconfig/network-scripts
# ls ifcfg-*
ifcfg-eth0  ifcfg-eth1  ifcfg-lo
In this example, there are two configuration files for Ethernet interfaces, ifcfg-eth0 and ifcfg-eth1, and one for the loopback interface, ifcfg-lo. The system reads the configuration files at boot time to configure the network interfaces.
The following are sample entries from an ifcfg-eth0 file for a network interface that obtains its IP address using the Dynamic Host Configuration Protocol (DHCP):
DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
USERCTL=no
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
HWADDR=08:00:27:16:C3:33
PEERDNS=yes
PEERROUTES=yes
If the interface is configured with a static IP address, the file contains entries such as the following:
DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
USERCTL=no
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
HWADDR=08:00:27:16:C3:33
IPADDR=192.168.1.101
NETMASK=255.255.255.0
BROADCAST=192.168.1.255
PEERDNS=yes
PEERROUTES=yes
The following configuration parameters are typically used in interface configuration files:
BOOTPROTO
How the interface obtains its IP address:
bootp
Bootstrap Protocol (BOOTP).
dhcp
Dynamic Host Configuration Protocol (DHCP).
none
Statically configured IP address.
BROADCAST
IPv4 broadcast address.
DEFROUTE
Whether this interface is the default route.
DEVICE
Name of the physical network interface device (or a PPP logical device).
HWADDR
Media access control (MAC) address of an Ethernet device.
IPADDR
IPv4 address of the interface.
IPV4_FAILURE_FATAL
Whether the device is disabled if IPv4 configuration fails.
IPV6_FAILURE_FATAL
Whether the device is disabled if IPv6 configuration fails.
IPV6ADDR
IPv6 address of the interface in CIDR notation. For example: IPV6ADDR="2001:db8:1e11:115b::1/32"
IPV6INIT
Whether to enable IPv6 for the interface.
MASTER
Specifies the name of the master bonded interface, of which this interface is slave.
NAME
Name of the interface as displayed in the Network Connections GUI.
NETMASK
IPv4 network mask of the interface.
NETWORK
IPV4 address of the network.
NM_CONTROLLED
Whether the network interface device is controlled by the network management daemon, NetworkManager.
ONBOOT
Whether the interface is activated at boot time.
PEERDNS
Whether the /etc/resolv.conf file used for DNS resolution contains information obtained from the DHCP server.
PEERROUTES
Whether the information for the routing table entry that defines the default gateway for the interface is obtained from the DHCP server.
SLAVE
Specifies that this interface is a component of a bonded interface.
TYPE
Interface type.
USERCTL
Whether users other than root can control the state of this interface.
UUID
Universally unique identifier for the network interface device.

Saturday, August 10, 2013

How to Configure Red Hat Cluster Services Fencing with iLO 3

Information
Environment :

Red Hat Cluster Suite 4+
Red Hat Enterprise Linux 5 Advanced Platform (Clustering)
Red Hat Enterprise Linux Server 6 (with the High Availability Add on)

Description :
Support for the iLO3 fence device has been added with the release of cman 2.0.115-34.el5_5.4 through erratum RHEA-2010-0876 which provides support for iLO3 via fence_ipmilan.
The iLO3 firmware should be a minimum of 1.15 as provided by HP.

Details
Resolution :
On both cluster nodes, install the following OpenIPMI packages used for fencing:
$ yum install OpenIPMI OpenIPMI-tools

Stop and disable the 'acpid' daemon:
$ service acpid stop; chkconfig acpid off

Start ipmi service on all cluster nodes:
$ service ipmi start; chkconfig ipmi on

Test ipmitool interaction with iLO3:
$ ipmitool -H -I lanplus -U -P chassis power status

The desired output is:
Chassis Power is on

Edit the /etc/cluster/cluster.conf to add the fence device:

<?xml version="1.0"?> <cluster alias="rh5nodesThree" config_version="32" name="rh5nodesThree"> <fence_daemon clean_start="0" post_fail_delay="1" post_join_delay="3"/> <clusternodes> <clusternode name="rh5node1.examplerh.com" nodeid="1" votes="1"> <fence> <method name="1"> <device domain="rh5node1" name="ilo3_node1"/> </method> </fence> </clusternode> <clusternode name="rh5node2.examplerh.com" nodeid="2" votes="1"> <fence> <method name="1"> <device domain="rh5node2" name="ilo3_node2"/> </method> </fence> </clusternode> <clusternode name="rh5node3.examplerh.com" nodeid="3" votes="1"> <fence> <method name="1"> <device domain="rh5node3" name="ilo3_node3"/> </method> </fence> </clusternode> </clusternodes> <cman expected_votes="3"> <multicast addr="229.5.1.1"/> </cman> <fencedevices> <fencedevice agent="fence_ipmilan" power_wait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3_node1" passwd="password"/> <fencedevice agent="fence_ipmilan" power_wait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3_node2" passwd="password"/> <fencedevice agent="fence_ipmilan" power_wait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3_node3" passwd="password"/> </fencedevices> <rm> <failoverdomains/> <resources/> </rm> </cluster>

Test that fencing is successful. From node1 attempt to fence node2 as follows:
$ fence_node node2

Wednesday, October 3, 2012

VSFTPD installation


Contents 

This tutorial is split up into the following topics:
introduction
installation
base configuration
xinetd vs. standalone
PAM configuration
creating virtual users (PAM)
virtual user configuration
Appendixes:
vsftpd configuration options
xinetd configuration options
faq


Introduction

This tutorial has actually being written because more and more people are trying to setup a ftp service, but mainly choose software with a bad security history like wu-ftpd for that task. My personal suggestion for a ftp server is vsftpd because of it's security, performance and stability.

We will be using virtual users here since they do not have real privileges - unlike real system users. For additional information please consult the faq.


Installation 
Before we can start with the real topic of this tutorial, we need to install vsftpd of course. Since we want to run vsftpd with virtual users and a per-user configuration we require at least version 1.1.0 of vsftpd. I have been using a backport of vsftpd 1.2.1-1 when writing this tutorial.

If your distribution is Debian/GNU Linux, you need to either backport it yourself or use my backport of vsftpd, since currently vsftpd 1.0.0-2 is in stable and 1.2.1-1 in testing. For other distributions you should check with your distribution if you can rely on a pre-built vsftpd.


Base configuration 
We will not start using a bloated standard configuration and adopt that to our needs - which is the way most people set up their services - no, we will configure from scratch changing the default values only where required:

/etc/vsftpd.conf (without chmod capabilities)

# =========================================================================
# base configuration
# -------------------------------------------------------------------------
anon_world_readable_only=NO
anonymous_enable=NO
chroot_local_user=YES
guest_enable=YES
guest_username=ftp
hide_ids=YES
listen=YES
listen_address=192.168.0.1
local_enable=YES
max_clients=100
max_per_ip=1
nopriv_user=ftp
pam_service_name=ftp
pasv_max_port=65535
pasv_min_port=64000
session_support=NO
use_localtime=YES
user_config_dir=/etc/vsftpd/users
userlist_enable=YES
userlist_file=/etc/vsftpd/denied_users
xferlog_enable=YES
# -------------------------------------------------------------------------


# =========================================================================
# ftp settings
# -------------------------------------------------------------------------
anon_umask=0027
async_abor_enable=YES
connect_from_port_20=YES
dirlist_enable=NO
download_enable=NO
# =========================================================================



The above configuration in combination with the default values of vsftpd provides a pretty secure default configuration, which we will then override on a per-user basis.

However if you require the capability to chmod then the above configuration will not work since this is not allowed for anonymous users. You should only use the configuration file below if you do really require chmod capabilites. You would require chmod capabilites for instance when the users should be able to change the permissions of "sensitive" information from the default umask you have specified in the per-user configuration.

/etc/vsftpd.conf (with chmod capabilities)

# =========================================================================
# base configuration
# -------------------------------------------------------------------------
anonymous_enable=NO
chroot_local_user=YES
guest_enable=YES
guest_username=ftp
hide_ids=YES
listen=YES
listen_address=192.168.0.1
local_enable=YES
max_clients=100
max_per_ip=1
nopriv_user=ftp
pam_service_name=ftp
pasv_max_port=65535
pasv_min_port=64000
session_support=NO
use_localtime=YES
user_config_dir=/etc/vsftpd/users
userlist_enable=YES
userlist_file=/etc/vsftpd/denied_users
virtual_use_local_privs=YES
xferlog_enable=YES
# -------------------------------------------------------------------------


# =========================================================================
# ftp settings
# -------------------------------------------------------------------------
async_abor_enable=YES
connect_from_port_20=YES
dirlist_enable=NO
download_enable=NO
local_umask=0027
# =========================================================================



Now we need to:

create directory /etc/vsftpd
create directory /etc/vsftpd/users
write list of denied users to /etc/vsftpd/denied_users
I suggest adding every system user in /etc/vsftpd/denied_users so no system user is asked for submisson of the password in plaintext. Use cat /etc/passwd | cut -d ":" -f 1 | sort > /etc/vsftpd/denied_users for creating that file.

Now when a user, who is listed in /etc/vsftpd/denied_users attempts to login, the session will be terminated before prompting for the password like illustrated below:

example ftp session for denied user

Connected to 192.168.0.1.
220 (vsFTPd 1.2.0)
Name (192.168.0.1:root): root
530 Permission denied.
Login failed.
ftp> quit
221 Goodbye.




xinetd vs. standalone
If you like to use the power of xinetd, for instance to restrict the usage of the ftp server to a specified time range or a couple of ip addresses, you can launch vsftpd from xinetd.

For that purpose you will require to change the base configuration, in detail remove the listen and listen_address configuration option and configure your xinetd service:

/etc/xinetd.d/ftp

service ftp
{
   banner_fail   = /etc/vsftpd/busy_banner
   disable      = no
   instances      = 100
   log_on_failure   += HOST
   log_on_success   += PID HOST DURATION
   no_access      = 192.168.0.3
   only_from      = 192.168.0.0/28
   per_source   = 2
   server      = /usr/sbin/vsftpd
   socket_type   = stream
   user      = root
   wait      = no
}



The above configuration will of course need to be adjusted for your needs, like you probably want to limit the number of concurrent sessions (instances) even more or ban a couple of subnetworks (no_access).

The banner_fail file could look like:

/etc/vsftpd/busy_banner
421 Server busy, please try again later!


PAM configuration
After providing the username and verifiying it is not contained in /etc/vsftpd/denied_users, we still can not login since we have nothing left to authentificate against left - assuming our /etc/vsftpd/denied_users always contains all usernames from /etc/passwd.

Therefore we now need to configure our real authentification which will be based upon PAM. As example we can authentificate against a username/password file in common database format:

/etc/pam.d/ftp

auth    required /lib/security/pam_userdb.so db=/etc/vsftpd/accounts
account required /lib/security/pam_userdb.so db=/etc/vsftpd/accounts




creating virtual users (PAM)
Before being able to login, we need to create a valid user. Depending on the PAM authentification backend this steps could vary. For instance when using a database as authentification backend you would require addind that user to the specified table.

If you would like to follow from the above PAM sample configuration, you will need the db_load program for creating the file in common database format. When using Debian just apt-get install libdb3-util. Afterwards you need to create a file which contains the login and on the next line the password:

sample accounts.tmp (for building accounts.db)
user1
password_for_user1
user2
password_for_user2


After creating the accounts.tmp, which is just a list of usernames and passwords, you need to build the database with db3_load -T -t hash -f accounts.tmp /etc/vsftpd/accounts.db. Afterwards you can erase your accounts.tmp since it is no longer required - until you upgrade your username/password database. You should now set pretty restrictive permissions to the database: chmod 600 /etc/vsftpd/accounts.db


Virtual user configuration 
Depending on your base configuration, you have a different per-user configuration:

/etc/vsftpd/users/user1 (without chmod capabilites)

anon_mkdir_write_enable=YES
anon_other_write_enable=YES
anon_upload_enable=YES
dirlist_enable=YES
download_enable=YES
local_root=/home/user1
write_enable=YES



If you require chmod capabilities and have specified that in your base configuration, you will go for the following:

/etc/vsftpd/users/user1 (with chmod capabilites)

dirlist_enable=YES
download_enable=YES
local_root=/home/user1
write_enable=YES




Appendix a: vsftpd configuration options
The configuration file takes a couple of options, which are partly explained shortly below. For more information please refer to the vsftpd.conf man page, where this information has been stripped off.

option description
anon_umask The value that the umask for file creation is set to for anonymous users.
anon_mkdir_write_enable If set to YES, anonymous users will be permitted to create new directories under certain conditions. For this to work, the option write_enable must be activated, and the anonymous ftp user must have write permission on the parent directory.
anon_other_write_enable If set to YES, anonymous users will be permitted to perform write operations other than upload and create directory, such as deletion and renaming. This is generally not recommended but included for completeness.
anon_upload_enable If set to YES, anonymous users will be permitted to upload files under certain conditions. For this to work, the option write_enable must be activated, and the anonymous ftp user must have write permission on desired upload locations.
anon_world_readable_only When enabled, anonymous users will only be allowed to download files which are world readable. This is recognising that the ftp user may own files, especially in the presence of uploads.
anonymous_enable Controls whether anonymous logins are permitted or not.
async_abor_enable When enabled, a special FTP command known as "async ABOR" will be enabled. Only ill advised FTP clients will use this feature. Addtionally, this feature is awkward to handle, so it is disabled by default. Unfortunately, some FTP clients will hang when cancelling a transfer unless this feature is available, so you may wish to enable it.
chroot_local_user If set to YES, local users will be placed in a chroot() jail in their home directory after login.
connect_from_port_20 This controls whether PORT style data connections use port 20 (ftp-data) on the server machine. For security reasons, some clients may insist that this is the case. Conversely, disabling this option enables vsftpd to run with slightly less privilege.
dirlist_enable If set to NO, all directory list commands will give permission denied.
download_enable If set to NO, all download requests will give permission denied.
guest_enable If enabled, all non-anonymous logins are classed as "guest" logins. A guest login is remapped to the user specified in the guest_username setting.
guest_username This setting is the real username which guest users are mapped to.
hide_ids If enabled, all user and group information in directory listings will be displayed as "ftp".
listen If enabled, vsftpd will run in standalone mode. This means that vsftpd must not be run from an inetd of some kind. Instead, the vsftpd executable is run once directly. vsftpd itself will then take care of listening for and handling incoming connections.
listen_address If vsftpd is in standalone mode, the default listen address (of all local interfaces) may be overridden by this setting. Provide a numeric IP address.
local_enable Controls whether local logins are permitted or not. If enabled, normal user accounts in /etc/passwd may be used to log in.
local_root This option represents a directory which vsftpd will try to change into after a local (i.e. non-anonymous) login. Failure is silently ignored.
local_umask The value that the umask for file creation is set to for local users.
max_clients If vsftpd is in standalone mode, this is the maximum number of clients which may be connected. Any additional clients connecting will get an error message.
max_per_ip If vsftpd is in standalone mode, this is the maximum number of clients which may be connected from the same source internet address. A client will get an error message if they go over this limit.
nopriv_user This is the name of the user that is used by vsftpd when it want to be totally unprivileged. Note that this should be a dedicated user, rather than nobody. The user nobody tends to be used for rather a lot of important things on most machines.
pam_service_name This string is the name of the PAM service vsftpd will use.
pasv_max_port The maximum port to allocate for PASV style data connections. Can be used to specify a narrow port range to assist firewalling.
pasv_min_port The minimum port to allocate for PASV style data connections. Can be used to specify a narrow port range to assist firewalling.
session_support This controls whether vsftpd attempts to maintain sessions for logins. If vsftpd is maintaining sessions, it will try and update utmp and wtmp. It will also open a pam_session if using PAM to authenticate, and only close this upon logout. You may wish to disable this if you do not need session logging, and you wish to give vsftpd more opportunity to run with less processes and / or less privilege.
use_localtime If enabled, vsftpd will display directory listings with the the time in your local time zone. The default is to display GMT. The times returned by the MDTM FTP command are also affected by this option.
user_config_dir This powerful option allows the override of any config option specified in the manual page, on a per-user basis. Usage is simple, and is best illustrated with an example. If you set user_config_dir=/etc/vsftpd_user_conf and then log on as the user "chris", then vsftpd will apply the settings in the file /etc/vsftpd_user_conf/chris for the duration of the session.
userlist_enable If enabled, vsftpd will load a list of usernames, from the filename given by userlist_file. If a user tries to log in using a name in this file, they will be denied before they are asked for a password. This may be useful in preventing cleartext passwords being transmitted.
userlist_file This option is the name of the file loaded when the userlist_enable option is active.
virtual_use_local_privs If enabled, virtual users will use the same privileges as local users. By default, virtual users will use the same privileges as anonymous users, which tends to be more restrictive (especially in terms of write access).
write_enable This controls whether any FTP commands which change the filesystem are allowed or not. These commands are: STOR, DELE, RNFR, RNTO, MKD, RMD, APPE and SITE.
xferlog_enable If enabled, a log file will be maintained detailling uploads and downloads.


Appendix b: xinetd configuration options
The xinetd configuration file takes a couple of different options, which are explained shortly below. For more information please refer to the xinetd.conf man page, where this information has been stripped off.

option description
banner_fail Takes the name of a file to be splatted at the remote host when a connection to that service is denied. This banner is printed immediately upon denial of access.

This is useful for informing your users that they are doing something bad and they shouldn't be doing it anymore.

disable This is boolean "yes" or "no".
This will result in the service being disabled and not starting.
instances Determines the number of servers that can be simultaneously active for a service (the default is no limit). The value of this attribute can be either a number or UNLIMITED which means that there is no limit.

log_on_failure Determines what information is logged when a server cannot be started (either because of a lack of resources or because of access control restrictions). The service id is always included in the log entry along with the reason for failure.

log_on_success Determines what information is logged when a server is started and when that server exits (the service id is always included in the log entry).

no_access Determines the remote hosts to which the particular service is unavailable. Its value can be specified in the same way as the value of the only_from attribute. These two attributes determine the location access control enforced by xinetd. If none of the two is specified for a service, the service is available to anyone. If both are specified for a service, the one that is the better match for the address of the remote host determines if the service is available to that host (for example, if the only_from list contains 128.138.209.0 and the no_access list contains 128.138.209.10 then the host with the address 128.138.209.10 can not access the service).

only_from Determines the remote hosts to which the particular service is available. Its value is a list of IP addresses which can be specified in any combination of the following ways:

a numeric address in the form of %d.%d.%d.%d. If the rightmost components are 0, they are treated as wildcards (for example, 128.138.12.0 matches all hosts on the 128.138.12 subnet). 0.0.0.0 matches all Internet addresses.
a factorized address in the form of %d.%d.%d.{%d,%d,...}. There is no need for all 4 components (i.e. %d.%d.{%d,%d,...%d} is also ok). However, the factorized part must be at the end of the address.
a network name (from /etc/networks).
a host name. When a connection is made to xinetd, a reverse lookup is performed, and the canonical name returned is compared to the specified host name. You may also use domain names in the form of .domain.com. If the reverse lookup of the client's IP is within .domain.com, a match occurs.
an ip address/netmask range in the form of 1.2.3.4/32.

per_source Takes an integer or "UNLIMITED" as an argument. This specifies the maximum instances of this service per source IP address.

server Determines the program to execute for this service
server_args Determines the arguments passed to the server.
socket_type Possible values for this attribute include:

dgram  datagram-based service
raw  service that requires direct access to IP
seqpacket  service that requires reliable sequential datagram transmission
stream  stream-based service

user Determines the uid for the server process. The user name must exist in /etc/passwd. This attribute is ineffective if the effective user ID of xinetd is not super-user.

wait This attribute determines if the service is single-threaded or multi-threaded. If its value is yes the service is single-threaded; this means that xinetd will start the server and then it will stop handling requests for the service until the server dies. If the attribute value is no, the service is multi-threaded and xinetd will keep handling new service requests.



Appendix c: faq
If you have a question regarding vsftpd which has not been answered in any section of this tutorial, feel free to email your question to website@linux-corner.net.

Frequently asked questions which have already been answered:

Firewalling information
Where can I find binary packages for other distributions than Debian?
Why do you prefer vsftpd over other ftp servers?
Why should one disallow system users?
Firewalling information
The configuration options pasv_max_port and pasv_min_port assist you in firewalling:
INPUT chain:
tcp
new/established
source port: 1024 - 65535
destination port: 21
tcp
new/established/related
destination port: pasv_min_port - pasv_max_port


OUTPUT chain:
tcp
related/established
source port: 20
destination port: 1024 - 65535
tcp
established
source port: 21
destination port: 1024 - 65535
tcp
established
source port: pasv_min_port - pasv_maxport
destination port: 1024 - 65535
With the above information (protocol, connection status, ports) you should be able to write the iptables ruleset. Your kernel needs to support connection tracking though; additionally you will require to use the ip_conntrack_ftp module of netfilter.

Where can I find binary packages for other distributions than Debian?
If you are not using Debian you are probably looking for RPM packages. I suggest you take a look at rpmseek.com. Here you can probably also find the db_load program used for creating virtual users with the common database format.
db3_load binary packages
vsftpd binary packages

Why do you prefer vsftpd?
I could quote that mostly from the vsftpd website. I am using vsftpd for it's excellent points in the following areas:
security
stability
performance
I am not really quoting it, I have verified all of the above points and I would not be using vsftpd if it would not perform that excellent. I know that there are other ftp servers out that are said to be secure, however until now I have not verified others.
Why should one disallow system users?
The typical system user has a lot of more privileges than required for the standard FTP user - like shell access. Granting each ftp user the privileges of a system user will definately affect system security, also since FTP transmitts passwords in cleartext. You can limit your real system users to not being able to login, however in that case there is no reason why you should not go for virtual users.

Immagine the root user logs in at the ftp server, the password is transmitted in cleartext and it is PRETTY easy to sniff out the password with standard tools. Do you really want to share passwords for users with probably unneccessary privileges with the whole world?


reference: http://www.debiansec.com/linux/services/ftp.html

Monday, October 1, 2012

Bash Parameter Expansion

If you use bash you already know what Parameter Expansion is, although you may have used it without knowing its name. Anytime you use a dollar sign followed by a variable name you're doing what bash calls Parameter expansion, eg echo $a or a=$b. But parameter expansion has numerous other forms which allow you to expand a parameter and modify the value or substitute other values in the expansion process.

Parameter expansion comes in many forms in bash, the simplest is just a dollar sign followed by a name, eg $a. This form merely substitutes the value of the variable in place of the parameter expansion expression. The variable name can also optionally be surround by braces, eg ${a}. If the variable name is immediately followed by characters that could be part of a variable name then the braces are needed to delimit the variable name, for example if you remove the braces from echo ${a}bc bash will try to expand the variable "abc" rather than "a".

One useful form of parameter expansion is to use a default value for a variable if it is not set. This is done with the syntax: ${VAR:-DFLT}. You might use this to allow your code to be modified via variables from the environment. Consider the following from a script, call it test.sh:

  TEST_MODE=${TEST_MODE:-0}
  ...
  if [[ $TEST_MODE -eq 0 ]]; then
      echo "Running in live mode"
  else
      echo "Running in test mode"
  fi

Normally the script runs in "live" mode but if you run it via:

  $ env TEST_MODE=1 sh test.sh

it runs in test mode.
You might also use the default value expansion with command line arguments or values from a config file, for example:

  # set cmd_param_x to 1 if seen on the command line
  ...
  if [[ ${cmd_param_x:-0} -eq 0 ]]; then
      echo "-x not specified"
  else
      echo "-x specified"
  fi

Another useful form of parameter expansion is to expand a variable and do string substitution on the value using the form ${VAR/search/replace}. For example:

  VAR=aabbcc
  echo ${VAR/b/-dd-}

outputs "aa-dd-bcc". Note that only the first instance of the search string is replaced, if you want to replace all instances use a double slash:

  VAR=aabbcc
  echo ${VAR//b/-dd-}

which now outputs "aa-dd--dd-cc".
There are also expansions for removing prefixes and suffixes. The form ${VAR#pattern} removes any prefix from the expanded value that matches the pattern. The removed prefix is the shortest matching prefix, if you use double pound-signs/hash-marks the longest matching prefix is removed. Similarily, the form ${VAR%pattern} removes a matching suffix (single percent for the shortest suffix, double for the longest). For example:

  file=data.txt
  echo ${file%.*}
  echo ${file#*.}

outputs the file base and extension respectively ("data" and "txt").
Note: if you have trouble remembering which is which of these two syntaxes, the "#" is to the left of the "%" key on your keyboard, just as prefixes come before suffixes. Also note that these are glob patterns not regular expressions.

Another expansion that exists is to extract substrings from the expanded value using the form ${VAR:offset:length}. This works in the expected form: offsets start at zero, if you don't specify a length it goes to the end of the string. For example:

  str=abcdefgh
  echo ${str:0:1}
  echo ${str:1}

outputs "a" and "bcdefgh".

This form also accepts negative offsets which count backwards from the end of the string. So this:

  str=abcdefgh
  echo ${str:-3:2}

produces "abcdefgh"... oops, what happened there? What happened was that bash misinterpretted what we wanted because the expansion looks like a default value expansion: ${VAR:-DFLT}. First time I tried this I stared at it for quite a while before a light came on as to how to do it (without using a variable [see below]):

  str=abcdefgh
  echo ${str:$((-3)):2}

which outputs the desired value "fg". The "$((...))" causes bash to treat the value as an arithmetic expansion (ie a number). Another slightly longer way of doing this is:

  str=abcdefgh
  i=-3
  echo ${str:$i:2}

The final form of parameter expansion I want to mention is one which simply expands to the length of the variable's value, its form is ${#VAR}. So for example:

  str=abcdef
  echo ${#str}

outputs "6".

Using these forms of parameter expansion in your shell scripts can simplify and shorten your scripts. These are not the only forms of parameter expansion that bash supports but they're the ones that I've found most useful over time. For more information see the "Parameter Expansion" section of the bash man page.

p.s. Note that all of the above forms of parameter expansion also work with bash's Special parameters: "$$", "$0", "$1", etc.

Credit to: Mitch Frazier

How to automatically backup mysql database using mysqldump?


It's a really good idea to use a least-privilege approach to most system administration tasks, and especially automated ones. This post describes using a "read only" MySQL user to handle backing up MySQL databases.

We use mysqldump to backup our databases on a regular basis, using scripts like this one:

#!/bin/sh

DIR=/backup/mysql/
DATESTAMP=$(date +%Y%m%d)
DB_USER=backup
DB_PASS='readonly'

# remove backups older than $DAYS_KEEP
DAYS_KEEP=30
find ${DIR}* -mtime +$DAYS_KEEP -exec rm -f {} \; 2> /dev/null

# create backups securely
umask 006

# list MySQL databases and dump each
DB_LIST=`mysql -u $DB_USER -p"$DB_PASS" -e'show databases;'`
DB_LIST=${DB_LIST##Database}
for DB in $DB_LIST;
do
  FILENAME=${DIR}${DB}-${DATESTAMP}.sql.gz
  mysqldump -u $DB_USER -p"$DB_PASS" --opt --flush-logs $DB | gzip > $FILENAME
done

You'll note that this script uses the user 'backup' to do the dumping. This is because our production servers grant potentially dangerous permissions (such as DROP TABLE) on a per-database basis. In order to run an automated backup, however, we need a single user that has just enough permissions to read from all the databases, but not enough to pose a risk to them.

The MySQL permissions required for the script above are SHOW DATABASES, SELECT, LOCK TABLES, and RELOAD. Grant them by entering the mysql command line and issuing these commands (choosing a better password than 'readonly' of course)

GRANT SHOW DATABASES, SHOW VIEW, SELECT, LOCK TABLES, RELOAD ON *.* to backup@localhost
 IDENTIFIED BY 'readonly';
FLUSH PRIVILEGES;

You can now back up all your databases by way of a single MySQL account that has just enough access to do the job, and not enough to cause significant harm. Which is what least-privilege access is all about.

Credit to: Stevem

Friday, September 28, 2012

How to force Sendmail to use smarthost without DNS

1. make sure your smarthost is in /etc/hosts.
2. Create /etc/mail/service.switch file and put following two lines inside:

hosts files
aliases files.

3.Replace this line in /etc/mail/submit.mc file.

define(`SMART_HOST',`my smarthost from /etc/hosts')dnl

4. Go to /etc/mail and type "m4 sendmail.mc > /etc/sendmail.cf" to build new sendmail.cf.
5. type /etc/init.d/sendmail restart

Friday, April 20, 2012

How to move MailArchiva and its data from one server to another


These instructions describe how to move MailArchiva and its data from one server to another

Directories#

There is no complex database replication procedure to perform. Copying MailArchiva from one machine to another is as simple as copying a few directories over to another machine (almost!).
(1) Install a clean copy of MailArchiva on the new server.
(2) Copy the following directories & files from the old server to the new server

Server Configuration#

C:\Program Files\MailArchiva\server\webapps\ROOT\WEB-INF\conf\* (windows)
/usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/conf/* (linux)

Logging Configuration#

C:\Program Files\MailArchiva\server\webapps\ROOT\WEB-INF\classes\log4j.xml (windows)
/usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/classes/log4j.xml

License File (EE only)#

C:\Program Files\MailArchiva\server\webapps\ROOT\WEB-INF\classes\license.lic (windows)
/usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/classes/license.lic

Audit & Log Files#

C:\Program Files\MailArchiva\server\logs\* (windows) /usr/local/mailarchiva/server/logs/* (linux)
C:\Program Files\MailArchiva\server\webapps\ROOT\WEB-INF\logs\* (windows)
/usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/* (linux)
C:\Program Files\MailArchiva\server\webapps\ROOT\WEB-INF\logs\index\* (windows)
/usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index* (linux)\

Volumes#

Copy all files in all volume store paths and index paths
(copying indexes over is optional)
Since there are potentially a large number of files in the store path, on Linux when using the cp command you may receive the errors "too many arguments"
To workaround this, follow one of the methods in the following article: http://www.linuxjournal.com/article/6060

Sunday, February 26, 2012

how to set default proxy server for linux

In file ~/.bash_profile, add this line.

http_proxy=[proxy_server:port]
ftp_proxy=[proxy_server:port]
export http_proxy
export ftp_proxy

how to set proxy for yum

In file /etc/yum.conf, add this line.

proxy=[proxy_server:port]
proxy_username=[proxy_username]
proxy_password=[password]

how to set proxy server for wget

In file /etc/wgetrc, edit this line.

http_proxy=[proxy_server:port]
ftp_proxy=[proxy_server:port]

Saturday, February 4, 2012

init: Id "x" respawning too fast: disabled for 5 minutes


In most Linux distributions this means that the system is booting by default into runlevel 5, which is supposed to respawn (re-start again after it’s been exited) a gui login via x windows, kdm, gdm, or whatever, but the system can’t locate the program. As a result below logs will keep appearing on your /var/log/messages for every 5 minutes.

Feb 4 08:55:25 server init: Id "x" respawning too fast: disabled for 5 minutes

However, “Id” can also indicate the absence or misconfiguration of another program, like mingetty, if init tries to respawn itself more than 10 times in 2 minutes.

Reason: You remove the “X Window System” and “GNOME Desktop Environment” but never modified the /etc/inittab.

Solution:

vi /etc/inittab

Change the initdefault id to 3 as below:

id:3:initdefault:

save and reboot your system


Or you may need to reinstall the “X Window System” and “GNOME Desktop Environment”.

yum groupinstall "X Window System"
yum groupinstall "GNOME Desktop Environment"

Clean up cache and log for magento


Create cleanup.php file in document root folder then put these code to it, edit database variables, save and run cleanup.php?clean=log or cleanup.php?clean=var


<?
$db['host'] = '';                    // Database hostname
$db['name'] = '';                    // Database name
$db['user'] = '';                    // Database username
$db['pass'] = '';                    // Database password
$db['pref'] = '';                    // Database tables' prefix

if($_GET['clean'] == 'log') clean_log_tables();
if($_GET['clean'] == 'var') clean_var_directory();

function clean_log_tables() {
    global $db;
 
    $tables = array(
        'log_url',
        'log_url_info',
        'log_visitor',
        'log_visitor_info'
    );
 
    mysql_connect($db['host'], $db['user'], $db['pass']) or die(mysql_error());
    mysql_select_db($db['name']) or die(mysql_error());
 
    foreach($tables as $v => $k) {
        mysql_query('TRUNCATE `'.$db['pref'].$k.'`') or die(mysql_error());
    }
}

function clean_var_directory() {
    $dirs = array(
        'var/cache/',
        'var/log/',
        'var/report/',
        'var/session/',
        'var/tmp/',
        'downloader/pearlib/cache/*',
        'downloader/pearlib/download/*'
    );
 
    foreach($dirs as $v => $k) {
        exec('rm -rf '.$k);
    }
}
?>



Reference: http://www.magentocommerce.com/boards/viewthread/36148/#t117754