Sunday, 14 July 2013

Useful linux commands

OS Details

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

System Information

$ uname -a
Linux p3307963.pubip.serverbeach.com 2.6.32-46-generic #108-Ubuntu SMP Thu Apr 11 15:56:25 UTC 2013 x86_64 GNU/Linux


$ uname -v

x86_64 (means system is 64 bit machine, otherwise its 32 bit machine)


User specific resource limit get/set

$ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Above command is very very important. Many a times TCP/IP socket connection grows a lot, if you sense that kind of problem, too many connections at a time, try to change "open files" limit to higher value, 20000 is  good enough. How to change the limit, you should read this article How to set ulimit in ubuntu/debian linux systems

Which user has opened how many files in sort order

$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 daemon

Check listening ports
$ netstat -nlp

change the open file limit in debian/ubuntu/linux system

For every user there is a resource limit configuration in linux. If it is not specified, it picks the default values, you can check them by command 
$ ulimit -a

Change Open file limit, ulimit on Debian/Ubuntu/Linux systems

------
Now to change it first of all pam limits by default is not loaded in ubuntu

$ vi /etc/pam.d/su
Un-comment the following line
#session    required   pam_limits.so
to
session    required   pam_limits.so

Now
$ vi /etc/security/limits.conf

and add the following lines to the end of the file (before the line # End of file)
*       soft  nofile   16000                                                      
*       hard nofile  64000

Save the file and quit vi.

Now To bring this in affect, you must restart the machine.

Below two lines will change the limit only for mysql user
mysql               soft    nofile          10240
mysql               hard   nofile          10240

Usually we need to change the "open file" limit from 1024(which is default) to higher value. But before that you should check which user is opening how many files.
$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 memcache
      4 daemon

The above result clearly says that root user has opened 1256 files, in such cases default limit of 1024 will start creating IO wait issue, connection timeout issue. To fix them, above article is beautiful.

------------

How to change Open file limit, ulimit on CentOS/Fedora/Red Hat


Command to check
ulimit -n
ulimit -a


1. vi sysctl.conf  and add this line  fs.file-max = 65536

2. vi /etc/security/limits.conf
*          soft     nproc          16384
*          hard     nproc          65535
*          soft     nofile         16384
*          hard     nofile         65535

3. Restart the server

4. ulmiit -n
16384


Mantis is not sending Email, How to fix?

I installed mantis for bug tracking system and it was done quite quick, may be in an hour. But the miserable part is, it was not sending email notification for any update, and it took more than couple of hour and still not fix... wtf is that.

OK, Chill !!!! I got the solution


Configuring the email settings is challenging task for most of the users start working with Mantis. The confusion is partially caused because PHP (so therefore Mantis) does not give you a very precise description about why it can not deliver emails. But you should try to see the log file /var/log/apache2/error.log and see if you can get some valuable information to proceed further.


In Most of the cases the configuration is very easy, and the good part is Mantis does maintain the mail related configuration variable as global ones.
So just need to change them and it should start working.


1. open your config_inc.php (located in: /var/www/mantis/config_inc.php)
2. copy the following code to the file:

$g_allow_signup    = ON;  //allows the users to sign up for a new account
$g_enable_email_notification = ON; //enables the email messages
$g_phpMailer_method = PHPMAILER_METHOD_SMTP; // this is most important
$g_smtp_host = 'smtp.gmail.com';
$g_smtp_connection_mode = 'tls';
$g_smtp_port = 587;
$g_smtp_username = 'youraccount@gmail.com'; //replace it with your gmail address
$g_smtp_password = '*********'; //replace it with your gmail password
$g_administrator_email = 'admin@example.com'; //this will be your administrator email address

3. go to your Mantis homepage (http://www.example.com/mantis)
4. click sign up for a new account
5. create a dummy account with your gmail address
6. press Signup
7. check your mail

--------------
Suppose you want to use other modes of smtp_connection, you can read about them at here http://www.mantisbt.org/docs/master-1.2.x/en/administration_guide/admin.config.email.html

If you want to use your own SMTP server without TLS or SSL conneciton, it would be something like.
$g_smtp_host = 'smtp.example.com';
$g_smtp_connection_mode = '';
$g_smtp_port = 25;
$g_smtp_username = 'youraccount@example'; //replace it with the email which can access your mail server
$g_smtp_password = '*********'; //replace it with your email account password

--------------
 /**
         * select the method to mail by:
         * PHPMAILER_METHOD_MAIL - mail()
         * PHPMAILER_METHOD_SENDMAIL - sendmail
         * PHPMAILER_METHOD_SMTP - SMTP
         * @global int $g_phpMailer_method
         */
        $g_phpMailer_method             = PHPMAILER_METHOD_MAIL;

Above code is present in "/mantis_home/config_defaults_inc.php" file.
As you see above mantis provide various ways of sending email, default is mail() command of linux, better is is make it "PHPMAILER_METHOD_SMTP", so that you can send email using any email server.

In the same file you can change from email also which will be shown in email. For that you need change the constant 
$g_from_email = "mantis@example.com"
$g_from_name = "Mantis Nishal Bug Tracker"

Note : For any further help, use this link http://www.mantisbt.org/forums/viewtopic.php?t=15398&f=3

#mantis not sending email notifications
#mantis email system not working
#mantis is not sending email
#mantis "sh: /usr/sbin/sendmail: not found"

Tuesday, 2 July 2013

Using compression, gizp, mod_deflate, amazon cloudfront issue with gzip

If you are not using compression, you are missing something great feature. Please enable it, you'll find a big difference in page load time.

How does gzip work over the HTTP ?


When browsers make a request to a server, they send a Accept-Encoding header.
For most of the browser they will send Accept-Encoding: gzip, deflate. The server then knows that this browser accepts data compressed using gzip or deflate. Now, the server sees Accept-Encoding: gzip, deflate, sends the response compressed as gzip and marks it with the response header Content-Encoding: gzip.

The server can also optionally send another header Vary: Accept-Encoding. This tells proxies to vary the object in the proxy cache based on the Accept-Encoding header. The result is that the proxy will have a compressed and uncompressed version of the file in cache (and maybe even three: uncompressed, gzip compressed, deflate compressed). Failing to provide the Vary header may result in the wrong encoding going to an incompatible browser. The Vary header was introduced in HTTP/1.1


How to use gzip/deflate?


With Apache, it comes with a module mod_deflate. 

In any standard installation of Apache it comes with mod deflate. If it is not enabled, you can enable using.
$ a2nmod defalte
And reload conf or restart apache. After enabling, check the configuration
/etc/apache2/mods-enabled/deflate.conf

<IfModule mod_deflate.c>
          # these are known to be safe with MSIE 6
          AddOutputFilterByType DEFLATE text/html text/plain text/xml

          # everything else may cause problems with MSIE 6
          AddOutputFilterByType DEFLATE text/css
          AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
          AddOutputFilterByType DEFLATE application/rss+xml
</IfModule>

Nginx comes by default with gzip module

edit the fle /etc/nginx/nginx.conf

          gzip on;
          gzip_disable "msie6";
          gzip_vary on;
          gzip_proxied any;
          gzip_comp_level 6;
          gzip_buffers 16 8k;
          gzip_http_version 1.1;
          gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

Issues with Amazon aws cloudfront and nginx and compression

If you are using cloudfront as a CDN to deliver the static content, then using nginx, you'll suddenly see that cloudfront is serving unzipped static contents.

The reason is cloudfront uses HTTP 1.0 to make request to origin server, but in nginx there is directive "gzip_http_version" which is set to 1.1, just check the above configuration. So what you need to do it make it 1.0

gzip_http_version 1.0;

That will enable the compression for cloudfront requests as well.

Monday, 1 July 2013

Install latest version of Nginx

$ sudo -s
$ echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
$ apt-get update

At this step you might get an error in upgrading
"W: GPG error: http://nginx.org lucid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ABF5BD827BD9BF62"

So try this
$ wget http://nginx.org/packages/keys/nginx_signing.key
$ cat nginx_signing.key | sudo apt-key add -
OK
$ apt-get update

Now Install
$apt-get install nginx

$/usr/sbin/nginx -V

Thursday, 27 June 2013

Redirect in nginx

From www.example.com to example.com


server {
         server_name     www.example.com
          return    301 $scheme://example.com$request_uri;
}

From example.com to www.example.com


server {
         server_name     example.com
          return    301 $scheme://www.example.com$request_uri;
}

-------
Some people try it in this way also, but it's a bad way of doing redirect as per nginx documentation. http://wiki.nginx.org/Pitfalls

server   {
   server_name www.domain.com;
   rewrite  ^/(.*)$  http://domain.com/$1 permanent;
}

server   {
   server_name domain.com;
   rewrite  ^/(.*)$  http://www.domain.com/$1 permanent;
}

Securing your website while using nginx, Deploying SSL certificates in nginx

Nginx is very very simple for deploying certificated and start serving HTTPS requests. Just create the copy of server block that you have written for serving HTTP requests and create another server block with the following changes.

server {
        server_name www.example.com;
        listen 443;
        ssl on;
        ssl_certificate      /etc/ssl/certs/www.example.com.crt;
        ssl_certificate_key  /etc/ssl/private/server.key;

        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;

        location ~* \.(jpg|jpeg|gif|css|png|js|ico|html|txt|pdf)$ {
                root /var/www;
                access_log off;
                expires 365d;
        }

        location / {
                proxy_pass        http://localhost:8181/;

                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwarded-Proto $scheme;
                add_header Front-End-Https   on;
        }
}

Issues and troubleshoots


1. While installing certificates, in the configuring you do not need to keep the intermediate certificate as you would have seen in Apache. Browsers usually store intermediate certificates which they receive and which are signed by trusted authorities, so actively used browsers may already have the required intermediate certificates and may not complain about a certificate sent without a chained bundle.
To check that try this URL : http://www.sslshopper.com/ssl-checker.html

To solve this possible issue : copy the intermediate certificate content in the main certificate content but after the main content.

$ cat bundle.crt >> www.example.com.crt

2. Here is a known error which you might face
"SSL_CTX_use_PrivateKey_file(" ... /www.example.com.key") failed (SSL: error:0B080074:x509 certificate routines: X509_check_private_key:key values mismatch)"

This error means "nginx has tried to use the private key to use the certificate" and you might have copied the intermediate certificate first and then main certificate content, because in that case private key will not match. So change the content on www.example.com.crt to have main content first and
then intermediate certificate contents.

$ cat main_certificate bundle.crt > www.example.com.crt

If that is the not the case, possibly you should check the certificate issuing authorities, because somehow private key is not matching. Or try to figure out by reading the log file "/var/log/nginx/error.log".

3. One of the most important thing is to add these lines in configuration
proxy_set_header X-Forwarded-Proto $scheme;
        add_header Front-End-Https   on;

Because when you do the proxy_pass you do it on http protocol, so even if user is making https request, your back-end server won't be aware of that. So pass that information in a X header, "X-Forwarded-Proto" is de-facto to pass the protocol information over proxies.

Correspondingly in tomcat,  if you are using JAVA based application, request.isSecure() will not work any more. So write a central API to get the
protocol information, something like this.

public static boolean isSecure(HttpServletRequest request){
String protocol=request.getHeader("X-Forwarded-Proto");
if("https".equals(protocol)){
return true;
}else{
return request.isSecure();
}
}