Monday, 4 March 2013

How to setup replication (Master Slave) in MySQL

I'll start the article by assuming that there are two MySQL server ready and we just need to do the configuration setup to start the replication.

Go to Master Server

1. Make all the tables engine = innodb 
As only innodb engines have binary logging feature which is essentially used for replication. Binary logging must be enabled on the master because the binary log is the basis for sending data changes from the master to its slaves. If binary logging is not enabled, replication will not be possible. MyIsam does not support binary logging.

Use following command to convert all the tables to InnoDB

mysql > SELECT CONCAT('ALTER TABLE ', table_name, ' ENGINE=InnoDB;') as ExecuteTheseSQLCommands
FROM information_schema.tables WHERE table_schema = 'db_name' 
ORDER BY table_name DESC;

2. Start binary logging on Master and Assign server Id (Server ID assigning is necessary, If you omit server-id (or set it explicitly to its default value of 0), a master refuses connections from all slaves).
edit the file /etc/mysql/my.cnf
[mysqld]
log-bin=mysql-bin
server-id=101

*For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1 and sync_binlog=1 in the master my.cnf file.

3. Create a slave user on Master DB
mysql> grant REPLICATION SLAVE on *.* to 'slave'@'IP_ADDRSS_OF_SLAVE' identified by 'slavePassword';
mysql> flush privileges;

4. Restart Master DB
and check 
mysql> show master status;

You can also check if mysql-bin log files are getting created on not, where you have given path of data to be stored, may be at /var/lib/mysql

5. Take a dump of database
 mysqldump -uroot -proot --single-transaction --master-data --databases db1,db2 > all_db.sql
And transfer the file on slave Machine

Go to slave Machine

6. Assign Server Id on slave DB
[mysqld]
server-id=102

7. Restart Slave DB

8. Import the dump file in database
mysql -uroot -proot < all_db.sql

9. Make this slave listen to Master
mysql> CHANGE MASTER TO MASTER_HOST='MASTE_HOST_IP_ADDRESS',MASTER_USER='slave',MASTER_PASSWORD='slavePassword';
mysql> flush privileges;

10. slave start
mysql > slave start;
mysql > show slave status;

DONE :)

Some troubleshoots and points:
  1. You can configure on slave mysql configuration that which all database or even which all tables you want to replicate or do not want to replicate
    http://dev.mysql.com/doc/refman/5.0/en/replication-options-slave.html
  2. If replication fails due to data consistency issue means, data already exist in slave and master is still trying to push (may be due to several kind of issue), you can change the slave configuration to move ahead
mysql > change MASTER TO MASTER_LOG_POS=desired_position;
mysql > change MASTER TO Master_Log_File='mysql-bin._desired_bin_log_file'

Reference : http://dev.mysql.com/doc/refman/5.0/en/replication-howto.html



Wednesday, 27 February 2013

Using POP on multiple clients or mobile devices


Using POP on multiple clients or mobile devices

If you have configured the email on outlook and on Blackberry/Android/iPhone/Gmail too, and on one of the client you are not able to receive the email, this article is useful to you.

Essentially POP (Post office protocol) is a one-way download of your messages that allows you to access your mail with a mail program like Outlook Express or Apple Mail. POP only offers one-way communication, which means that actions you take in the mail program. You should know two things "recent mode" and "Leave a copy of message on server".

What is 'recent mode?'
If you're accessing Gmail on multiple clients through POP, Gmail's 'recent mode' makes sure that all messages are made available to each client, rather than only to the first client to access new mail.
Recent mode fetches the last 30 days of mail, regardless of whether it's been sent to another POP1 client already.
Setting up 'recent mode'
In your POP client settings, replace 'username@gmail.com' in the 'Username' or 'Email' field with 'recent:username@gmail.com'
Once you enable recent mode, please be sure to configure your POP client to leave messages on the server according to the instructions below:
  • Outlook or Outlook Express: on the Advanced tab, check the box next to 'Leave a copy of messages on the server.'
  • Apple Mail: on the Advanced tab, remove the check next to 'Remove copy from server after retrieving a message.'
  • Thunderbird: on the Server Settings tab, check the box next to 'Leave messages on server.'

* This is an exact copy the URL https://support.google.com/mail/bin/answer.py?hl=en&answer=47948

Friday, 22 February 2013

Updating and installing package on debian machine


Its very simple :)
APT (Advance Package Tool) is a free user interface which is used in debian machine to install/remove/update any software. For the same in Red Hat machine is "yum".

How to search a package and Install?

Points :

  1. $ apt-cache search "Is the command to search a package"
  2. $ apt-get install "Is the command to install a package"
  3. apt-cache - query the APT cache
  4. apt-cache search/madison are two important commands you should know
  5. $ apt-cache search - "Performs a full text search on all available package lists for the POSIX regex pattern given"
  6. $ apt-cache madison - "Command attempts to mimic the output format and a subset of the functionality of the Debian archive management tool, madison. It displays available versions of a package in a tabular format"
  7. sudo apt-get install =version - "Is the command to install a package with a certain version"
  8. dpkg -s - "Is the command to about the package"


Example :

$ apt-cache search mysql-server
cacti - Frontend to rrdtool for monitoring systems and services
phpbb2-conf-mysql - Automatic configurator for phpbb2 on MySQL database
torrentflux - web based, feature-rich BitTorrent download manager
mysql-server - MySQL database server (meta package depending on the latest version)
mysql-server-5.0 - MySQL database server binaries

$ apt-cache madison mysql-server
mysql-server | 5.0.96-0ubuntu3 | http://us.archive.ubuntu.com hardy-security/main Packages
mysql-server | 5.0.96-0ubuntu3 | http://us.archive.ubuntu.com hardy-updates/main Packages
mysql-server | 5.0.51a-3ubuntu5 | http://us.archive.ubuntu.com hardy/main Packages

Now if you want to install a certain version package, you should use
$ apt-get install mysql-server=5.0.96-0ubuntu3

Command to know about the package
$ dpkg -s mysql-server

References
1. http://manpages.ubuntu.com/manpages/natty/man8/apt-cache.8.html
2. http://manpages.ubuntu.com/manpages/hardy/man8/apt-get.8.html




Sunday, 17 February 2013

Mysql Database Configuration, Access settings, Innodb configuration, Log slow query


To bind the server Access point
-----------------------------------
By default it is binded to localhost or 127.0.0.1
Open /etc/mysql/my.cnf fine

So lets say you have 4-5 machines from which you want to access mysql DB from any of the machine, but you do not want anyone from outside to access this,
bind-address will be Local LAN Ip Address.
bind-address = Local LAN IP Address

If you want only local machine to access mysql DB
bind-address            = 127.0.0.1

If you want to make it public, remove the bind-address line.

Logging the slow queries
-----------------------------
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 1


Logging the queries which are not using index
---------------------------------------------------
log-queries-not-using-indexes


Changing the InnoDB configuration
----------------------------------------
Buffer Pool Size is the memory which you provide to mysql server program.

innodb_buffer_pool_size=5120M
innodb_lock_wait_timeout=20
innodb_rollback_on_timeout

max_allowed_packet : 
The max_allowed_packet variable limits the size of a single result set. In the [mysqld] section, any normal connection can only get that much worth of data in a single query. In mysqldump you typically produce "extended INSERT" queries, where you list multiple rows within the same INSERT command. It's better, then, to have this variable set high. In mysqld max_allowed_packet could be 16M (to be safe, because it doesn't uses memory until required), in mysqldump, max_allowed_packet  could be 128M or may be 512M, depends on your machine and requirement.

If you want mysqldump to work fast
---------------------------------------

[mysqldump]
quick
quote-names
max_allowed_packet  =  64M (Increase this value, default is 16M)

* You can also take take dump faster by passing as a command argument
$ mysqldump -u root -p --max_allowed_packet=512M dbname > dbname.sql

More Ideas on MySQL performance tuning
1. https://blogs.oracle.com/luojiach/entry/mysql_innodb_performance_tuning_for
2. http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/


Wednesday, 16 January 2013

After setting javaagent in classpath tomcat is not starting



Q. I am trying to use newrelic as javaagent in tomcat on ubuntu machine. When I set the javaagent in classpath, tomcat fails to start. I tried with tomcat5 and tomcat7, its not starting. And I am not getting any log also to check anything.
I tried doing the same with tracelytics javaagent, still its the same, tomcat is not starting. And clue, no log, someone please help.
A. It was happening due to setting of Xss configuration at the time server start in CATALINA_OPTS. I had set it to 128K, somehow it was failing to start. But when I removed that setting, it started, then I changed to 256K then also it started. I am not sure about the inside story, but at least it is working now :)

Friday, 7 December 2012

What to do when you get a new server?


First of all have a big smile mmmmmmmmmmmmmm :)

It's always been a lovely experience to play with a new server, it's like you got a new power source and you start thinking of how to optimize the utilization of the resource, Shall i create distributed memory system or I shall deploy some other components on this machine because it is having higher computation power, woooow. It's fascinating.

Any ways, I am first writing this basic set of commands which will work on Debian systems (and yes Ubuntu is built on Debian architecture), to set the basic things always required on a new machine.

First thing you must do is
$ apt-get update
$ apt-get upgrade

1. Using command apt-cache you can search the package 
apt-cache search jdk
2. Using command apt-get you can install the package
$ apt-get install openjdk-6-jdk
$ apt-get install mysql-server
$ apt-get install atop

If you want only mysql client - try this
$apt-get install mysql-client-5.5

3. Set the timezone on debian machine
$ dpkg-reconfigure tzdata
And choose Asia->Kolkata

4. Installing sar on ubuntu
$ sudo apt-get install sysstat
$ sudo vi /etc/default/sysstat
change ENABLED=”false” to ENABLED=”true”
$ sudo vi /etc/cron.d/sysstat
Change 5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
To */2 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

$ sudo service sysstat restart
$ sar -A
If you want to save the statistics for further analysis to a file use:
$ sudo sar -A > $(date +`hostname`-%d-%m-%y-%H%M.log)

5. apt-get install zip

6. You might want to change the file limits - find details at here
http://nishal-tech.blogspot.in/2013/07/how-to-set-ulimit-in-ubuntudebian-linux.html



Thursday, 13 September 2012

ActiveMQ Detected missing/corrupt journal files

If you are getting an error while starting ActiveMQ something like this

2012-09-13 18:23:39,823 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Detected missing/corrupt journal files. 1 messages affected. | org.apache.activemq.broker.BrokerService | main
java.io.IOException: Detected missing/corrupt journal files. 1 messages affected.
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:626)
at org.apache.activemq.store.kahadb.MessageDatabase$6.execute(MessageDatabase.java:460)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:458)
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:315)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:357)
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:222)
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:180)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:186)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:485)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)

---
solution is :
Sometime kahaDB which is used by activeMQ to persist the message, get either corrupted some message/journal gets corrupted or missing. I got few flag which can can configured in activemq.xml.

   <persistenceAdapter> 
            <kahaDB directory="${activemq.base}/data/kahadb" 
                    ignoreMissingJournalfiles="true" 
                    checkForCorruptJournalFiles="true" 
                    checksumJournalFiles="true" />
     </persistenceAdapter> 

http://activemq.apache.org/kahadb.html

After setting these flag, start the activemq and watch it print something like this :


2012-09-13 18:33:26,144 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
2012-09-13 18:33:26,145 | INFO  | Recovery replayed 1 operations from the journal in 0.326 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2012-09-13 18:33:29,580 | INFO  | Some journal files are missing: [42658] | org.apache.activemq.store.kahadb.MessageDatabase | main
2012-09-13 18:33:29,774 | INFO  | Detected missing/corrupt journal files.  Dropped 1 messages from the index in 3.59 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main

Then you are done. If it still does not work. Remove the kahadb/db.data and restart, it will parse the journal to rebuild the index.

I am using 5.4.2 version of ActiveMQ, kahaDB version 3.

:)