APACHE - failed to start

[On]

Red Hat Enterprise Linux Server release 7.4 (Maipo)

Apache was not running and attempt to start the httpd service failed. The natural thing as a sysadmin you would check the error_log. 

# systemctl restart httpd

# tail -f /var/log/httpd/error_log (location could be different on your installation)

--- error snippet---

[Tue Feb 20 17:57:32.667940 2018] [auth_digest:error] [pid 6256] (28)No space left on device: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.6256

[Tue Feb 20 17:57:32.667948 2018] [auth_digest:error] [pid 6256] (28)No space left on device: AH01760: failed to initialize shm - all nonce-count checking, one-time nonces, and MD5-sess algorithm disabled

--- error snippet---

2 things to observe here. 

1. no space left of device
2. failed to create shared memory

As it turns out /run partition was used up 100% - which explains the subsequent error -  no. 2. 

Basically - the issue here is Apache has been leaving some stuff behind in shared memory. So the next command list those stuffs 

# ipcs -s (s is semaphores)

--- output ---

------ Semaphore Arrays --------

key        semid      owner      perms      nsems

0x00000000 3375104    apache     600        1

0x00000000 3407873    apache     600        1

0x00000000 2359298    apache     600        1

0x00000000 3440643    apache     600        1

0x00000000 3473412    apache     600        1

0x00000000 3506181    apache     600        1

--- output ---

You may proceed to remove the semaphores (apache only) by executing this command

# ipcrm -s  {semid} (if possible one after another)

Then proceed with restarting the httpd service

# systemctl restart httpd

# systemctl status httpd (verify Apache is running)

# ipcs -s (verify the semaphores are cleared)

Also, it is a good practice to reboot your machine once in 2 or 3 months that will actually clear the unused semaphores. 

PostgreSQL - Step by step guide installation from source

[Tested On]
CentOS Linux release 7.4.1708 (Core)

# wget https://ftp.postgresql.org/pub/source/v9.6.6/postgresql-9.6.6.tar.gz
# tar xvzf postgresql-9.6.6.tar.gz
# cd postgresql-9.6.6
# mkdir /opt/postgresql-9.6.6
# ./configure --prefix=/opt/postgresql-9.6.6
--- possible error output ---
configure: error: readline library not found
--- possible error output ---
# yum install readline-devel (install the missing library as reported above)
# make
# make install
# adduser postgres
# passwd postgres (set password)
# mkdir /opt/postgresql-9.6.6/data
# chown postgres:postgres /opt/postgresql-9.6.6/data
# su - postgres
$ /opt/postgresql-9.6.6/bin/initdb -D /opt/postgresql-9.6.6/data/
$ /opt/postgresql-9.6.6/bin/postmaster -D /opt/postgresql-9.6.6/data >/opt/postgresql-9.6.6/data/log 2>&1 &
$ ps -eaf | grep postgres (verify postgres is running)
$ vi ~/.bashrc
export PATH=$PATH:/opt/postgresql-9.6.6/bin
-- Save the file
-- Exit to root

--- Start and Stop postgresql using systemctl (systemd) ---
# cd /usr/lib/systemd/system
# vi postgres.service

--- Insert the following script  (change the postgresql location paths accordingly) ---
# START SCRIPT
[Unit]
Description=PostgreSQL database server
After=network.target

[Service]
Type=forking

User=postgres
Group=postgres

# Where to send early-startup messages from the server (before the logging
# options of postgresql.conf take effect)
# This is normally controlled by the global default set by systemd
# StandardOutput=syslog

# Disable OOM kill on the postmaster
OOMScoreAdjust=-1000
# ... but allow it still to be effective for child processes
# (note that these settings are ignored by Postgres releases before 9.5)
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0

# Maximum number of seconds pg_ctl will wait for postgres to start.  Note that
# PGSTARTTIMEOUT should be less than TimeoutSec value.
Environment=PGSTARTTIMEOUT=270

Environment=PGDATA=/opt/postgresql-9.6.6/data


ExecStart=/opt/postgresql-9.6.6//bin/pg_ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT}
ExecStop=/opt/postgresql-9.6.6/bin/pg_ctl stop -D ${PGDATA} -s -m fast
ExecReload=/opt/postgresql-9.6.6/bin/pg_ctl reload -D ${PGDATA} -s

# Give a reasonable amount of time for the server to start up/shut down.
# Ideally, the timeout for starting PostgreSQL server should be handled more
# nicely by pg_ctl in ExecStart, so keep its timeout smaller than this value.
TimeoutSec=300

[Install]
WantedBy=multi-user.target
# END SCRIPT

--- Save the file

# systemctl daemon-reload
# systemctl enable postgres
# systemctl start postgres

--- Verify postgres is running
# systemctl status postgres

LINUX - Installing a package after a fresh installation

[Tested On]

CentOS Linux release 7.4.1708 (Core) x64

I installed CentOS on my VM with "minimal selection" which later I came to realize that I required cc, gcc and python compilers to install a couple of third party software. 

I performed these steps :

# yum group list

The required compilers are available in the "Development Tools" package which you will find under the Available Groups:

To proceed with the installation, execute the following command:

# yum --setopt=group_package_types=mandatory,default,optional groupinstall "Development Tools"

This will basically install "all" packages under the Development Tools which I prefer to avoid any dependencies issues later on. 

LINUX - Delete user account and its home directory

[Tested On]

CentOS Linux release 7.4.1708 (Core)

First please verify user home directory 

# cat /etc/passwd | grep {user}

testuser:x:1001:1002::/home/testuser:/bin/bash

--- /home/testuser - is the user home directory

To delete the user and everything in its home directory, issue the command below

# userdel --remove test

LINUX - Test ping ipv6 address

[Tested On]
Red Hat Enterprise Linux Server release 7.4 (Maipo)

--- Obtain your ipv6 address and network device information

# ifconfig

--- Test ping using ping6

# ping6 {ipv6-address}%{network-device}

SMTP - Testing SMTP communication using telnet client

[Tested On]
Red Hat Enterprise Linux Server release 7.4 (Maipo)

Note: For Windows Server, please make sure telnet client is installed.

# telnet
open {mail-server} 25
EHLO {mail-server}
MAIL FROM:{sender}
RCPT TO:{recipient}
DATA
Subject: Test from SMTP Mail Relay
This is a test message
. (this is a dot)
QUIT

--- expected output ---
open {mail-server} 25
EHLO {mail-server}
-- output --

250-mail-server Hello mail-server [IP], pleased to meet you

250-ETRN

250-AUTH LOGIN CRAM-MD5 PLAIN

250-8BITMIME

250-ENHANCEDSTATUSCODES

250 SIZE
-- output --
MAIL FROM:{sender}
-- output --
250 2.1.0 Sender OK
-- output --
RCPT TO:{recipient}
-- output --
250 2.1.5 Recipient OK
-- output --
DATA
-- output --

Start mail input; end with <CLRF>.<CLRF>

-- output --

Subject: Test from SMTP Mail Relay

This is a test message
.

250 2.6.0 Ok, message saved

QUIT

221 2.0.0 See ya in cyberspace

Connection closed by foreign host.

--- expected output ---

LINUX - Restrict sftp user to home directory

In some cases, you are required to restrict sftp users ( using FileZilla or Winscp ) to their respective home directories. This is how you do it.

# useradd cent
# passwd cent
-- set passsword

# groupadd sftp_users
# usermod -G sftp_users cent

# vi /etc/ssh/sshd_config
-- line 147: comment out the line below
#Subsystem sftp /usr/libexec/openssh/sftp-server

-- add the following just after the above line
Subsystem sftp internal-sftp

-- add follows to the end
Match Group sftp_users
  X11Forwarding no
  AllowTcpForwarding no
  ChrootDirectory /home
  ForceCommand internal-sftp

-- Save the file

# systemctl restart sshd

Try to access using your favourite windows sftp client or using sftp command line :

# sftp cent@localhost

SSH - Configuring SSH with auto login (without a password)

[Tested On]

CentOS Linux release 7.4.1708 (Core)

Setup ssh operations between 2 or more linux machines without the need to enter password. 

* Note that the following steps should be executed for each separate user account. In this case, I am using the root account.

On machine-1
# ssh-keygen -t rsa
Condition: Enter file in which to save the key (/root/.ssh/id_rsa): press <ENTER>
Condition: Enter passphrase (empty for no passphrase): press <ENTER>
Condition: Enter same passphrase again: press <ENTER>

# ssh-copy-id -i ~/.ssh/id_rsa.pub root@machine-2
Condition: Are you sure you want to continue connecting (yes/no)? type yes <ENTER>
Condition: root@machine-2's password: type {root password} <ENTER>

# chmod 0600 ~/.ssh/authorized_keys

On machine-2
# ssh-keygen -t rsa
Condition: Enter file in which to save the key (/root/.ssh/id_rsa): press <ENTER>
Condition: Enter passphrase (empty for no passphrase): press <ENTER>
Condition: Enter same passphrase again: press <ENTER>

# ssh-copy-id -i ~/.ssh/id_rsa.pub root@machine-1
Condition: Are you sure you want to continue connecting (yes/no)? type yes <ENTER>
Condition: root@machine-1's password: type {root password} <ENTER>

# chmod 0600 ~/.ssh/authorized_keys

After completing the setup, you will now be able to ssh between machine 1 and 2 without a prompt for password.

SSL - Obtain information about your SSL configuration

# openssl s_client -connect localhost:443

SSH - Remove SSH weak ciphers

CBC and Arcfour based ciphers are considered no longer secure. Please follow the steps to remove them from the ssh server

# vi /etc/ssh/shh_config
Replace #Ciphers line with: Ciphers aes128-ctr,aes192-ctr,aes256-ctr
Replace #MACs line with: MACs hmac-sha1,umac-64@openssh.com,hmac-ripemd160
- Save the file

Note there will no longer be a # in front of Cyphers and MACs.

# vi /etc/ssh/shhd_config
Look for the line "# Ciphers and keying" and "#RekeyLimit default none"
Below "#RekeyLimit default none" add:
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
MACs hmac-sha1,umac-64@openssh.com,hmac-ripemd160
- Save the file

Restart sshd service
# systemctl restart sshd


To Test Weak Ciphers is no longer allowed :
# ssh -vv -oCiphers=aes128-cbc,3des-cbc,blowfish-cbc,aes256-arcfour,arcfour256,arcfour128 {target machine-ip}


To Check Current Allowed Ciphers on your localhost 
# sshd -T | grep "\(ciphers\|macs\)"

APACHE - Accessing multiple domain names using port 80

If you plan to publish 2 or more domain urls from the same apache server, this configuration will help you but the requirement is the document root path should be different for all domains.

<VirtualHost *:80>
       ServerName my-server1.com
       DocumentRoot /var/www/html/my-app1
    <Directory /var/www/html/my-app1>
Options FollowSymLinks
AllowOverride All
    Require all granted
    </Directory>
</VirtualHost>


<VirtualHost *:80>
       ServerName my-server2.com
       DocumentRoot /var/www/html/my-app2
    <Directory /var/www/html/my-app2>
Options FollowSymLinks
AllowOverride All
    Require all granted
    </Directory>
</VirtualHost>

<VirtualHost *:80>
       ServerName my-server3.com
       DocumentRoot /var/www/html/my-app3
    <Directory /var/www/html/my-app3>
Options FollowSymLinks
AllowOverride All
    Require all granted
    </Directory>
</VirtualHost>

APACHE - Reverse Proxy using root (/)

This took me days to figure out. I am happy to share with you guys who are wondering how to setup a reverse proxy. 

<VirtualHost *:8081>
       ServerName my-server
       DocumentRoot /var/www/html/my-app
       RewriteEngine On
       <Directory /var/www/html/my-app/>
      Options FollowSymLinks
              AllowOverride All
      Require all granted
       </Directory>
ProxyRequests Off
ProxyPreserveHost On

<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://{internal-app-server-ip}/
ProxyPassReverse / http://{internal-app-server-ip}/

</VirtualHost>

Save it and restart apache service.

When you access http://my-server:8081 in a browser apache will interpret the configuration above and redirects the url to http://{internal-app-server-ip}/ - internally while preserving the originating url in your browser which is http://my-server:8081. 

APACHE - Redirect URL containing an IP address to domain name

[On]
Red Hat Enterprise Linux Server release 7.4 (Maipo)

Please insert the Rewrite conditions as follows in the .htaccess file located in your DocumentRoot directory.

--- Change the IPs accordingly. ---

<IfModule mod_rewrite.c>
    <IfModule mod_negotiation.c>
        Options -MultiViews
    </IfModule>

    RewriteEngine On

    RewriteCond %{HTTP_HOST} ^10\.24\.19\.85$ [OR]
    RewriteCond %{HTTP_HOST} ^10\.24\.19\.109$
    RewriteRule ^(.*)$ https://{DOMAIN_NAME}/$1 [L,R=301]
</IfModule>

LINUX - Booting Procedure

How Linux Boot?
1. BIOS
2. MBR
3. GRUB
4. Kernel
5. Init
6. Runlevel

System Startup 


How computer startup?

• Booting is a bootstrapping process that starts operating systems when the user turns on a computer system.
• A boot sequence is the set of operations the computer performs when it is switched on that load on operating system.

Booting sequence

1. Power on

2. CPU jump to address of BIOS (0xFFFF0)

3. BIOS runs POST (Power on Self Test)

4. Find bootable devices

5. Loads and execute boot sector from MBR

6. Loads OS

BIOS

• BIOS refers to the software code run by a computer when first powered on.
• The primary function of BIOS is a code program embedded on a chip that recognises and controls various devices that make up the computer.

Boot Loader

MBR (Master Boot Record)

• OS is booted from a hard disk, where the Master Boot Record (MBR) contains the primary boot loader.
• The MBR is a 512-byte sector, located in the first sector on the disk (sector 1 of cylinder0, head 0)
• After the MBR is loaded into RAM, the BIOS yields control to it
• The first 446 bytes are the primary boot loader, which contains both executable code and error message text
• The next sixty-four bytes are the partition table, which contains a record for each of four partitions.
• The MBR ends with two bytes that are defined as the magic number (0xAA55). The magic number serves as a validation check on the MBR.

Extracting the MBR

• To see the contents of MBR, use this command:
• dd if=/dev/hda of=/mbr.bin bs=512 count=1
• od -xa mbr.bin

** The dd command, which needs to be run from root, reads the first 512 bytes from /dev/hda (IDE) /dev/sda (SCSI) and writes them to the mbr.bin file

** The od command prints the binary file in hex and ASCII formats

Boot loader could be more aptly called the kernel loader. The task at this stage is to load the Linux kernel. GRUB and LILO are the most popular Linux boot loader.

GRUB: Grand Unified Bootloader

• GRUB is an operating system independent boot loader
• A multiboot software packet from GNU
• Flexible command line interface
• File system access 
• Support multiple executable format
• Support diskless system
• Download OS from network,  etc...

GRUB boot process

1. This BIOS finds a bootable device (hard disk) and transfers control to the master boot record

2. The MBR contains GRUB stage 1. Given the small size of the MBR, Stage 1 just load the next stage of GRUB

3. GRUB Stage 1.5 is located in the first 30 kb of hard disk immediately following the MBR. Stage 1.5 loads Stage 2

4. GRUB Stage 2 receives control and displays to the user the GRUB boot menu (where the user can manually specify the boot parameters)

5. GRUB loads the user-selected (or default) kernel into memory and passes control on to the kernel

Kernel

Kernel Image

• The kernel is the central part in most computer operating systems because of it's task, which is the management of the system's resources and the communication between hardware and software components
• Kernel is always store on memory until computer is turn off
• Kernel image is not an executable kernel, but a compress kernel image
• zImage size less than 512 kb
• bzImage size greater than 512 kb

Task of Kernel

• Process management
• Memory management
• Device management
• System call

Init process

• The first thing the kernel does is to execute init program
• Init is the root/parent of all processes executing on Linux
• The first process that init starts is a script /etc/rc.d/rc.sysinit
• Based on the appropriate run-level, scripts are executed to start various processes to run the system and make it functional

The Linux Init Processes

• The init process is identified by process id "1"
• Init is responsible for starting system processes as defined in the /etc/inittab file
• Init typically will start multiple instances of "getty" which waits for console logins which spawn one's user shell process
• Upon shutdown, init controls the sequence and processes for shutdown

System Processes

Process ID 0 - The scheduler

Process ID 1 - The init process

Process ID 2 - kflushd

Process ID 3 - kupdate

Process ID 4 - kpiod

Process ID 5 - kswapd

LINUX - How can I get my external router IP?

curl -s http://checkip.dyndns.org/ | cut -d ' ' -f 6 | sed s/"body\|html\|<\|>\|\/"//g

or just visit http://www.whatismyip.com/

LINUX - su: incorrect password

The problem is that "su" has to be suid in order to read from the shadow file of your system to verify the password you have given. 

By issuing "chmod 777 *" you deleted the suid bit (777 = 0777). 

You have to reset this bit by executing "chmod 4755 /bin/su".

LINUX - Listing today's files only

# ls -l --time-style=+%D | grep `date +%D`

LINUX - netstat

1. Using Netstat to Count TCP Connection States
netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c

2. List TCP/UDP and it's PID
netstat -tuvnap

APACHE - Redirect http requests to https

First make sure Apache is configured for HTTPS connection and necessary SSL certificates are installed.

No non-ssl access i.e. only accept https connections

Now open httpd.conf or .htaccess file (mod_rewrite not required):
# vi httpd.conf

Append following line :
Redirect permanent / https://mail.nixcraft.com/

Any request made to http://mail.nixcraft.com will goto https://mail.nixcraft.com/

Save and close the file. Restart the Apache:
# systemctl restart httpd

LINUX - How to Tell 32 or 64-bit?

# getconf LONG_BIT