Parsing and Analyzing Apache Log file using linux commands

This content has been moved to a new URL.

Click here to read the actual article

5 step process to improve your SEO in a month

SEO (Search Engine Optimization) is something which is bit intangible and unpredictable and so quite interesting area also. Lots of people think to do the SEO but hardly anyone understand what they want to achieve out of SEO.

 Lets try to understand the intent and possible achievement out of SEO.

SEO can be understood as marketing strategy, by ranking up high in Google, but it's the only 1st step of SEO. SEO has 2 more steps left which has almost remained hidden.
So what are the steps:

1. Short term strategy - Helps in quick marketing, few keyword ranking for increasing your reach-ability, estimated time is 1 month
2. Long term strategy - Helps in building a sustainable traffic, estimated time is 6 months
3. Ecosystem strategy - Helps search engineer in refining their knowledge and search, estimated time is 2 years and more

Now a days the search engine algorithm is mature, intelligent, and intuitive. While there are quite a few things that require technical expertise, the overall theme of SEO is pretty simple. So if you are targeting something short term strategy and planning to do some keyword ranking optimization, it's pretty simple.

Do these 5 step process and it would improve SERP within a month.

1. High performance is must : Your page should load as fast as possible, As an average measurement in GA if it is taking more than 7 seconds, you are bad, 4 seconds is awesome, even if it is 5 seconds, it's good. Write good HTML to save DOM execution time, use CDN to deliver CSS, JS and Image, use caching etc.

2. Understand your business and customers and write a quality blog about educating your product, talking about peripheral of your business domain, product and may be your competitors. This helps a lot to Search engine to decide your seriousness of the product, their social need and as the people will start consuming your product, Search engine will give lot of weight to your product, your website. Blog is to educate them, providing extra value about your product. Google algorithm also considers social websites, blogs and social contents to be pushed into Google news section, which is an advantage.

3. Keep watching your spamy links : Google Penguin updates are targeting to penalized website who create spamy link backs. With October Penguin 3.0 update it's much more clear that any anchor text manipulation and spamy link will get penalized. It's good to keep watching your links through webmaster or some tool and audit them at least once in 6 month. And work towards get good quality link back, blog is one of the strategy to get good link backs.

4. Content is the king and play on keyword density: Creating high quality content is still the central idea in SEO. Think about creating a content in much more organized, creative and new style. Like providing good meta information, images, bread crumbs for navigation, readable and something new(be creative). Use the keyword density keyword repetition intelligently on the page. You can use SEO quake kind of tool to check the keyword density. Every keyword repetition should have a value, it should not be just stuffed.

5. Promote your website on social website
Create Facebook page, twitter handle and other social properties to talk about your product, gather people and promote your website, links. This has 2 advantage, one social index score becomes better which also plays a role in SEO and other you can get extra traffic from social media.

And Few small things to remember:

1. Don't publish or build business on illegal stuffs
2. HTTPS serving websites are getting extra value
3. Don't play on aggregation and content farming, now it doesn't work
4. Only keyword stuffing can kill you
5. Being mobile friendly website is quite helpful to receive mobile traffic

It's still simple to achieve good but you've have to focused, consistent and dedicated. It works, it really works :)

Some concepts of Solr

This article has been moved to a different location.
Click here to read the entire article

Enabling JMX in Tomcat

Any Java program can be monitored using Jconsole. Jconsole is an user interface which comes by default with any JDK package, you just need to type jconsole on your terminal and it will show you the interface. Jconsole works along with JMX, which has to be enabled in the respective java program or JVM.

How to enable JMX in tomcat?

You to enable these variables while starting of the JVM. In tomcat you can simple enable them by using CATALINA_OPTS variable in setenv.sh file in bin folder of tomcat.

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port={port to access} 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname={optional, allow what ip to access this Tomcat}

export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname="

To know more details about variables, you can check here,
http://tomcat.apache.org/tomcat-6.0-doc/monitoring.html

After changing this, start the tomcat and then run jconsole from anywhere and connect using
"IP:port", if authentication is enable, enter the credentials also. Now you can see all the JVM stuff, threads, memory usages, CPU usages, GCs performed etc.

RSA server certificate CommonName (CN) 'www.domain.com' does NOT match server name

Hi,

I was getting this error on restarting of my Apache,

"I receive the Error: "RSA server certificate CommonName (CN) 'www.domain.com' does NOT match server name"?"

And there was on website which was able to tell me that is something wrong, which is http://www.sslshopper.com/ssl-checker.html

In my case, I had missed an entry of "ServerName" directive in Apache virtual host configuration.

<VirtualHost _default_:443>
        ServerAdmin webmaster@domain.com
        ServerName www.domain.com

SSLEngine on
SSLCertificateFile /root/ssl/domain.crt
SSLCertificateKeyFile /root/ssl/server.key
SSLCertificateChainFile /root/ssl/bundle.crt

</VirtualHost>

So after entering the ServerName, apache error.log file stopped showing the error message. And the website sslshopper also, started getting the certificate of diagnosis, which made me believe, that issue is resolved.

In general, you should also check
1. DNS entry (host name and IP is correct)
2. /etc/hosts file
3. while creating CSR file, did you miss the common name by any chance, In this case create a new CSR and Re-key the certificate and deploy new certificates.
4. Check in Apache or Nginx, ServerName must match, certificates are issued for a fix domain.

fb:share_button/fb:login_button/fb: failed to resize in 45s

I saw one strange issue while using fb_share button which I would like to share and How did I fix it.

When you use fb_share buttons with their stats, means you are using some javascript to get the starts and show beside of the button. When the page load it was working fine, but when you navigate to any other page by clicking on a link or button, and you do browser back to the same page again, it tries to load fb_share button again.. but it keep the textarea locked for 45 seconds, which was quite strange.

I guess it had to be with while fb tries to get the data it also tries to make the proper UI of the button, size and all, and during that period, it locks the textarea.

I started searching for the issue and got a pathetic solution on stackoverflow which says to use this css

.fb-share-button span,
.fb-share-button iframe {
    width: 120px! important;
    height: 25px! important;
}

And amazing part is, it works :)

Make an anchor tag with href but do not let user navigate away

This can be utilized a lot for SEO purposes. Anchor tag href would be crawled by search engines, but when user will be click, it will not navigate away the user from the page, and you can handle the onclick even in anyway you want to.

<a href="http://www.google.com" onclick="dothis(event); return false;">Click me</a>

<script>
function dothis(e){
if (!e) var e = window.event;
alert("ok");
if (e.stopPropagation)
e.stopPropagation();
return false;
}
</script>

Useful linux commands

OS Details

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

System Information

$ uname -a
Linux p3307963.pubip.serverbeach.com 2.6.32-46-generic #108-Ubuntu SMP Thu Apr 11 15:56:25 UTC 2013 x86_64 GNU/Linux

$ uname -v

x86_64 (means system is 64 bit machine, otherwise its 32 bit machine)

User specific resource limit get/set

$ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Above command is very very important. Many a times TCP/IP socket connection grows a lot, if you sense that kind of problem, too many connections at a time, try to change "open files" limit to higher value, 20000 is  good enough. How to change the limit, you should read this article How to set ulimit in ubuntu/debian linux systems

Which user has opened how many files in sort order

$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 daemon

Check listening ports
$ netstat -nlp

change the open file limit in debian/ubuntu/linux system

For every user there is a resource limit configuration in linux. If it is not specified, it picks the default values, you can check them by command 
$ ulimit -a

Change Open file limit, ulimit on Debian/Ubuntu/Linux systems

------
Now to change it first of all pam limits by default is not loaded in ubuntu

$ vi /etc/pam.d/su
Un-comment the following line
#session    required   pam_limits.so
to
session    required   pam_limits.so

Now
$ vi /etc/security/limits.conf

and add the following lines to the end of the file (before the line # End of file)
*       soft  nofile   16000                                                      
*       hard nofile  64000

Save the file and quit vi.

Now To bring this in affect, you must restart the machine.

Below two lines will change the limit only for mysql user
mysql               soft    nofile          10240
mysql               hard   nofile          10240

Usually we need to change the "open file" limit from 1024(which is default) to higher value. But before that you should check which user is opening how many files.
$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 memcache
      4 daemon

The above result clearly says that root user has opened 1256 files, in such cases default limit of 1024 will start creating IO wait issue, connection timeout issue. To fix them, above article is beautiful.

------------

How to change Open file limit, ulimit on CentOS/Fedora/Red Hat

Command to check
ulimit -n
ulimit -a


1. vi sysctl.conf  and add this line  fs.file-max = 65536

2. vi /etc/security/limits.conf
*          soft     nproc          16384
*          hard     nproc          65535
*          soft     nofile         16384
*          hard     nofile         65535

3. Restart the server

4. ulmiit -n
16384

Mantis is not sending Email, How to fix?

I installed mantis for bug tracking system and it was done quite quick, may be in an hour. But the miserable part is, it was not sending email notification for any update, and it took more than couple of hour and still not fix... wtf is that.

OK, Chill !!!! I got the solution


Configuring the email settings is challenging task for most of the users start working with Mantis. The confusion is partially caused because PHP (so therefore Mantis) does not give you a very precise description about why it can not deliver emails. But you should try to see the log file /var/log/apache2/error.log and see if you can get some valuable information to proceed further.


In Most of the cases the configuration is very easy, and the good part is Mantis does maintain the mail related configuration variable as global ones.
So just need to change them and it should start working.


1. open your config_inc.php (located in: /var/www/mantis/config_inc.php)
2. copy the following code to the file:

$g_allow_signup    = ON;  //allows the users to sign up for a new account
$g_enable_email_notification = ON; //enables the email messages
$g_phpMailer_method = PHPMAILER_METHOD_SMTP; // this is most important
$g_smtp_host = 'smtp.gmail.com';
$g_smtp_connection_mode = 'tls';
$g_smtp_port = 587;
$g_smtp_username = 'youraccount@gmail.com'; //replace it with your gmail address
$g_smtp_password = '*********'; //replace it with your gmail password
$g_administrator_email = 'admin@example.com'; //this will be your administrator email address

3. go to your Mantis homepage (http://www.example.com/mantis)
4. click sign up for a new account
5. create a dummy account with your gmail address
6. press Signup
7. check your mail

--------------

Suppose you want to use other modes of smtp_connection, you can read about them at here http://www.mantisbt.org/docs/master-1.2.x/en/administration_guide/admin.config.email.html

If you want to use your own SMTP server without TLS or SSL conneciton, it would be something like.

$g_smtp_host = 'smtp.example.com';
$g_smtp_connection_mode = '';
$g_smtp_port = 25;
$g_smtp_username = 'youraccount@example'; //replace it with the email which can access your mail server
$g_smtp_password = '*********'; //replace it with your email account password

--------------

 /**

         * select the method to mail by:

         * PHPMAILER_METHOD_MAIL - mail()

         * PHPMAILER_METHOD_SENDMAIL - sendmail

         * PHPMAILER_METHOD_SMTP - SMTP

         * @global int $g_phpMailer_method

         */

        $g_phpMailer_method             = PHPMAILER_METHOD_MAIL;

Above code is present in "/mantis_home/config_defaults_inc.php" file.

As you see above mantis provide various ways of sending email, default is mail() command of linux, better is is make it "PHPMAILER_METHOD_SMTP", so that you can send email using any email server.

In the same file you can change from email also which will be shown in email. For that you need change the constant 

$g_from_email = "mantis@example.com"

$g_from_name = "Mantis Nishal Bug Tracker"

Note : For any further help, use this link http://www.mantisbt.org/forums/viewtopic.php?t=15398&f=3

#mantis not sending email notifications

#mantis email system not working

#mantis is not sending email

#mantis "sh: /usr/sbin/sendmail: not found"

Using compression, gizp, mod_deflate, amazon cloudfront issue with gzip

If you are not using compression, you are missing something great feature. Please enable it, you'll find a big difference in page load time.

How does gzip work over the HTTP ?

When browsers make a request to a server, they send a Accept-Encoding header.
For most of the browser they will send Accept-Encoding: gzip, deflate. The server then knows that this browser accepts data compressed using gzip or deflate. Now, the server sees Accept-Encoding: gzip, deflate, sends the response compressed as gzip and marks it with the response header Content-Encoding: gzip.

The server can also optionally send another header Vary: Accept-Encoding. This tells proxies to vary the object in the proxy cache based on the Accept-Encoding header. The result is that the proxy will have a compressed and uncompressed version of the file in cache (and maybe even three: uncompressed, gzip compressed, deflate compressed). Failing to provide the Vary header may result in the wrong encoding going to an incompatible browser. The Vary header was introduced in HTTP/1.1

How to use gzip/deflate?

With Apache, it comes with a module mod_deflate. 

In any standard installation of Apache it comes with mod deflate. If it is not enabled, you can enable using.
$ a2nmod defalte
And reload conf or restart apache. After enabling, check the configuration
/etc/apache2/mods-enabled/deflate.conf

<IfModule mod_deflate.c>
          # these are known to be safe with MSIE 6
          AddOutputFilterByType DEFLATE text/html text/plain text/xml

          # everything else may cause problems with MSIE 6
          AddOutputFilterByType DEFLATE text/css
          AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
          AddOutputFilterByType DEFLATE application/rss+xml
</IfModule>

Nginx comes by default with gzip module

edit the fle /etc/nginx/nginx.conf

          gzip on;
          gzip_disable "msie6";
          gzip_vary on;
          gzip_proxied any;
          gzip_comp_level 6;
          gzip_buffers 16 8k;
          gzip_http_version 1.1;
          gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

Issues with Amazon aws cloudfront and nginx and compression

If you are using cloudfront as a CDN to deliver the static content, then using nginx, you'll suddenly see that cloudfront is serving unzipped static contents.

The reason is cloudfront uses HTTP 1.0 to make request to origin server, but in nginx there is directive "gzip_http_version" which is set to 1.1, just check the above configuration. So what you need to do it make it 1.0

gzip_http_version 1.0;

That will enable the compression for cloudfront requests as well.

Install latest version of Nginx

$ sudo -s
$ echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
$ apt-get update

At this step you might get an error in upgrading
"W: GPG error: http://nginx.org lucid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ABF5BD827BD9BF62"

So try this
$ wget http://nginx.org/packages/keys/nginx_signing.key
$ cat nginx_signing.key | sudo apt-key add -
OK
$ apt-get update

Now Install
$apt-get install nginx

$/usr/sbin/nginx -V

Redirect in nginx

From www.example.com to example.com

server {
         server_name     www.example.com
          return    301 $scheme://example.com$request_uri;
}

From example.com to www.example.com

server {
         server_name     example.com
          return    301 $scheme://www.example.com$request_uri;
}

-------
Some people try it in this way also, but it's a bad way of doing redirect as per nginx documentation. http://wiki.nginx.org/Pitfalls

server   {
   server_name www.domain.com;
   rewrite  ^/(.*)$  http://domain.com/$1 permanent;
}

server   {
   server_name domain.com;
   rewrite  ^/(.*)$  http://www.domain.com/$1 permanent;
}

Securing your website while using nginx, Deploying SSL certificates in nginx

Nginx is very very simple for deploying certificated and start serving HTTPS requests. Just create the copy of server block that you have written for serving HTTP requests and create another server block with the following changes.

server {
        server_name www.example.com;
        listen 443;
        ssl on;
        ssl_certificate      /etc/ssl/certs/www.example.com.crt;
        ssl_certificate_key  /etc/ssl/private/server.key;

        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;

        location ~* \.(jpg|jpeg|gif|css|png|js|ico|html|txt|pdf)$ {
                root /var/www;
                access_log off;
                expires 365d;
        }

        location / {
                proxy_pass        http://localhost:8181/;

                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwarded-Proto $scheme;
                add_header Front-End-Https   on;
        }
}

Issues and troubleshoots

1. While installing certificates, in the configuring you do not need to keep the intermediate certificate as you would have seen in Apache. Browsers usually store intermediate certificates which they receive and which are signed by trusted authorities, so actively used browsers may already have the required intermediate certificates and may not complain about a certificate sent without a chained bundle.
To check that try this URL : http://www.sslshopper.com/ssl-checker.html

To solve this possible issue : copy the intermediate certificate content in the main certificate content but after the main content.

$ cat bundle.crt >> www.example.com.crt

2. Here is a known error which you might face
"SSL_CTX_use_PrivateKey_file(" ... /www.example.com.key") failed (SSL: error:0B080074:x509 certificate routines: X509_check_private_key:key values mismatch)"

This error means "nginx has tried to use the private key to use the certificate" and you might have copied the intermediate certificate first and then main certificate content, because in that case private key will not match. So change the content on www.example.com.crt to have main content first and
then intermediate certificate contents.

$ cat main_certificate bundle.crt > www.example.com.crt

If that is the not the case, possibly you should check the certificate issuing authorities, because somehow private key is not matching. Or try to figure out by reading the log file "/var/log/nginx/error.log".

3. One of the most important thing is to add these lines in configuration
proxy_set_header X-Forwarded-Proto $scheme;
        add_header Front-End-Https   on;

Because when you do the proxy_pass you do it on http protocol, so even if user is making https request, your back-end server won't be aware of that. So pass that information in a X header, "X-Forwarded-Proto" is de-facto to pass the protocol information over proxies.

Correspondingly in tomcat,  if you are using JAVA based application, request.isSecure() will not work any more. So write a central API to get the
protocol information, something like this.

public static boolean isSecure(HttpServletRequest request){
String protocol=request.getHeader("X-Forwarded-Proto");
if("https".equals(protocol)){
return true;
}else{
return request.isSecure();
}
}

nginx proxy_pass configuration, complexity, settings, issues, solutions

Ideally when you set these parameters for proxy_pass, its good enough.

location / {
               proxy_pass        http://localhost:8080;
               proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             
                proxy_connect_timeout      30;
                proxy_send_timeout         30;
                proxy_read_timeout         600;

                proxy_buffer_size          4k;
                proxy_buffers              4 16k;
                proxy_busy_buffers_size    64k;
                proxy_temp_file_write_size 64k;
}

How to pass the remote address to back-end server while using nginx

In case of proxy_pass, there is a complexity, when back-end server will try to access the requested IP address it will return either 127.0.0.1 or may be the local subnet IP where nginx is deployed, because nginx is proxy server and it overrides the information of requested IP address. So the solution is to set an extra parameters in request header at the time of making proxy.
statement "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;" meant for that only.

How to get the remote address in your back-end server while using "X-Forwarded-For"

Story is not yet over, Not now you have to do something at your back-end server to extract the requested IP from request header.

In case of back-end is Apache it is little simple, you just need to install a module
$ sudo apt-get install libapache2-mod-rpaf
And configure the file /etc/apache2/mods-available/rpaf.conf

<IfModule mod_rpaf.c>
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
</IfModule>

But in case of back-end as Tomcat, it is little complex, 

you will never get it via request.getRemoteAddr(); So write a global API to access the remote address, like this,

public static String getRemoteAddress(HttpServletRequest request){

String ip = request.getHeader("X-Forwarded-For");

if(ip==null || "".equals(ip)){

ip=request.getRemoteAddr();

}

return ip;

}

So if it is found in "X-Forwarded-For" as a request headers, it will return or else it will get from request.getRemoteAddr(). This kind of programming is good because tomorrow if you plan to use Apache proxying using AJP protocol, then you don't need to make any back-end change, in case of AJP, it will get you remote address directly from request object, and in case of other proxying, it will get you from "X-Forwarded-For" header.

Some other configurations points

1. proxy_connect_timeout directive assigns a timeout for the connection to the upstream server(or back-end server). It's default value is 60s.

This is not the time until the server returns the pages, that is the proxy_read_timeout statement. If your upstream server is up, but hanging (e.g. it does not have enough threads to process your request so it puts you in the pool of connections to deal with later), then this statement will not help as the connection to the server has been made. 

So in case you ever get proxy_connect_timeout at nginx, check your back-end connection limit.

2. proxy_read_timeout - this is very very important, default value is 60s.

This directive sets the read timeout for the response of the proxied server. It determines how long nginx will wait to get the response to a request. The timeout is established not for entire response, but only between two operations of reading.

In contrast to proxy_connect_timeout, this timeout will catch a server that puts you in it's connection pool but does not respond to you with anything beyond that, then proxy_read_timeout will come in picture. Be careful though not to set this too low, as your proxy server might take a longer time to respond to requests on purpose (e.g. when serving you a report page that takes some time to compute). 

You can also set different proxy_read_timeout which could be higer value like 10minutes for certain location.

location /admin/reports/ {

// other proxy_pass settings

proxy_read_timeout  600;

}

location / {

// other proxy_pass settings

proxy_read_timeout  30;

}

3. proxy_send_timeout - default value is 60s

This directive assigns timeout with the transfer of request to the upstream server. Timeout is established not on entire transfer of request, but only between two write operations. If after this time the upstream server will not take new data, then nginx is shutdown the connection.

Nginx setup for segregating static and dynamic content from nginx and back-end server using proxy_pass

This configuration will set the static contents to be served from nginx and dynamic contents from back-end server, may be Apache (in case of PHP based application), Tomcat (in case of Java based application).

For this purpose we essentially use proxy_pass module of nginx. Its very very simple with nginx, create two different location context and serve them differently. Once using the root mean providing the directory where to find the file, and other use proxy_pass

server {
          server_name www.example.com;
 
location ~* \.(jpg|jpeg|gif|css|png|js|ico|html)$ {
root /var/www;
access_log off;
expires 365d;
}

location / {
 proxy_pass        http://localhost:8181;
 proxy_set_header Host $host;
                  proxy_set_header X-Real-IP $remote_addr;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Now here come some basic knowledge of nginx

1. In nginx you can set as many locations as possible, what ever is best match will be picked and executed.
2. If you want to create something like "www.example.com/static/" and entire URL after /static/ should be served from nginx only, you can do that.
location /static/ {
         root /var/www/static/;
         access_log off;
         expires 30d;
}

3. "access_log off" means, it will not create any log record for such request which match that location.

4. "expires 30d" means, it will set expiry header to 30 days for all such requests which will match that location. Like in apache we use mod_expires for setting expiration time of the static contents, so that browser can cache that contents for a long time. In nginx its just a one line, :)

5.  proxy_pass will let you forward the request to any back-end server.
"proxy_pass        http://localhost:8080;" will forward your request for dynamic contents possibly to back-end tomcat server.

#How to set expiration time for static contents while using nginx

nginx - (13: Permission denied) while reading upstream

2013/06/07 21:13:38 [crit] 17799#0: *717313 open() "/var/lib/nginx/proxy/2/19/0000000192" failed (13: Permission denied) while reading upstream, client: 122.167.17.4, server: www.example.com, request: "GET /web/jsp/example.jsp HTTP/1.1", upstream: "http://127.0.0.1:8181//web/jsp/example.jsp", host: "www.example.com"

Typically this is a problem of saving the buffered data from the proxy server and sending it back. When the upstream server response returns large number of bytes then nginx keeps the part of data at the disk and start sending the first received bytes to browser. So for that it uses a certain directory to maintain the data at the configured directory, which is "/var/lib/nginx/proxy" in my case. So you just need to give access to that directory to worker user of nginx.

1. open the /etc/nginx/nginx.conf to find worker user of nginx
2. Or ps -ef | grep nginx  and check which user is running worker process
$ ps aux | grep “nginx: worker process” | awk ‘{print $1}’
www-data
3. In my case it is www-data
4. Give access to that directory
$ chown -R www-data.www-data /var/lib/nginx
5. Done


Nginx access related articles Why nginx usually throws 403, Forbidden?

# Nginx is returning on part data in response
# Nginx response is chunked abnormally

Configure an alternate JAVA

Get the new JDK and put it in /usr/liv/jvm and run this commands

$ update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk-7u21/bin/java" 1
$ update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk-7u21/bin/javac" 1
$ update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk-7u21/bin/javaws" 1

$ chmod a+x /usr/bin/java
$ chmod a+x /usr/bin/javac
$ chmod a+x /usr/bin/javaws
$ chown -R root:root /usr/lib/jvm/jdk-7u21

Make sure you read the below command output clear and choose which Java you want.
$ update-alternatives --config java
There are 2 alternatives which provide `java'.

  Selection    Alternative
-----------------------------------------------
 +        1    /usr/lib/jvm/java-6-openjdk/jre/bin/java
*         2    /usr/lib/jvm/jdk-7u21/bin/java

Press enter to keep the default[*], or type selection number: (If you want java 6, write 1 and if you java 7 write 2)

* This commands are being good in debian/ubuntu system. I am not sure about Red Hats/CentOs

Load Balancing using Apache mod_jk

After installing apache, install mod_jk module.

1. $ apt-get install libapache2-mod-jk

2. create a file jk.conf(if not present) in mods-available directory and write these lines
JkWorkersFile   /etc/apache2/workers.properties
JkLogFile       /var/log/apache2/mod_jk.log
JkShmFile       /var/log/apache2/mod_jk.shm
JkLogLevel      error

After creating file, may be you need to do symbol linking to mods-enabled directory. Or just disabling and enabling the mod_jk will do.

$ a2dismod jk
$ a2enmod jk

3. create a file /etc/apache2/workers.properties
and write lines for load balancing, creating a load balancer and workers and assembling the workers in load balancer. In below configuration server1 will take 60% load and server2 will take 40%.
#
worker.list=loadbalancer

worker.server1.port=8009 (-- The product server port where you want to forward the request)
worker.server1.host=server1 IP Address (-- The proxy server IP where you want to forward the request)
worker.server1.type=ajp13 (-- Protocol setting, in case of mod_jk module, it will be always ajp13)
worker.server1.lbfactor=60 (-- This parameter to set, how much load this server1 will be given)

worker.server2.port=8009
worker.server2.host=server2 IP Address
worker.server2.type=ajp13
worker.server2.lbfactor=40

worker.loadbalancer.type=lb
worker.loadbalancer.sticky_session=true
worker.loadbalancer.balance_workers=server1,server2

4. In virtual host configuration of Apache
In below example, all the request starting with /web/static/css(js)(images), will be JkUnMounted so , they will be served from apache document root directory(which is /var/www ), and rest all requests will be JkMounted, so they will be forwarded to load balancer and load balancer will forward the reuqest either to server1 or server2. This is done using AJP protocol, so make sure that you have configured AJP protocol in your application server like tomcat. TO configure the tomcat, you can check this url Configuration Apache and Tomcat to user Mod_jk connector for proxy passing


  ServerAlias www.example.com
  DocumentRoot /var/www
  ServerName example.com

  DocumentRoot /var/www/

  JkMount /* loadbalancer
  JkUnMount /web/static/css/* loadbalancer
  JkUnMount /web/static/js/* loadbalancer
  JkUnMount /web/static/images/* loadbalancer

  ErrorLog /var/log/apache/example-com-error_log
  CustomLog /var/log/apache/example-com-access_log combined

Apache performance tuning and security tuning

MaxKeepAliveRequests

It's actually the maximum number of requests to serve on a TCP connection. If you set it up to 100, clients with keepalive support will be forced to reconnect after downloading 100 items. Default in Apache is 100, you can increase it if you have enough memory on the system. If you are serving a page which contain high number of images then keeping is high is better because then it utilize the alive connections to serve the image requests.

KeepAliveTimeout

KeepAliveTimeout determines how long to wait for the next request. Set this to a low value, perhaps between two to five seconds. If it is set too high, child processed are tied up waiting for the client when they could be used for serving new clients.

MaxRequestsPerChild

The MaxRequestsPerChild directive sets the limit on the number of requests that an individual child server process will handle. After MaxRequestsPerChild requests, the child process will die. It's set to 0 by default, the child process will never expire. It is appropriate to set this to a value of few thousands. This can help prevent memory leakage, since the process dies after serving a certain number of requests. Don't set this too low, since creating new processes does have overhead.

Proper user of MPM (Multi-Processing Module)

This I have already explain at this URLConfiguring Apache/Tomcat for serving Maximum number of requests

Security tweaks

1. ServerTokens
This directive configures what you return as the Server HTTP response
Header. The default is 'Full' which sends information about the OS-Type and compiled in modules.
# Set to one of:  Full | OS | Minimal | Minor | Major | Prod
where Full conveys the most information, and Prod the least, you can also set it to "ProductOnly" which is best

ServerTokens ProductOnly

2. ServerSignature
Optionally add a line containing the server version and virtual host
# Set to one of:  On | Off | EMail
You can Set to "EMail" to also include a mailto: link to the ServerAdmin, better to set it to Off

ServerSignature Off

3. TraceEnable 
This Allow TRACE method to enable/disabled
# Set to one of:  On | Off | extended
Set to "extended" to also reflect the request body, best it to make it Off

TraceEnable Off