{"id":1084,"date":"2011-12-08T08:49:48","date_gmt":"2011-12-08T08:49:48","guid":{"rendered":"http:\/\/truelogic.org\/wordpress\/?p=1084"},"modified":"2014-05-13T05:27:57","modified_gmt":"2014-05-13T05:27:57","slug":"complete-solution-to-migrate-a-large-database-sql-file-in-mysql","status":"publish","type":"post","link":"https:\/\/truelogic.org\/wordpress\/2011\/12\/08\/complete-solution-to-migrate-a-large-database-sql-file-in-mysql\/","title":{"rendered":"Complete solution to migrate a large database sql file in MySql"},"content":{"rendered":"

\"php<\/a>During my recent project, i was in a situation where I had to move a large mysql database from a testing server to live server. I got stuck almost on every phase of this whole situation. Firstly it created problems in exporting the database from testing server (phpmyadmin has many issues with large sql file handling, I guess) and when I was importing on live server it again threw me many problems.<\/p>\n

The purpose of this blog is to tell others how to overcome all these steps and how to move a large database using different methods. This method is not the only method, but was my personal choice. Many experienced users have used other alternative solutions for doing this stuff.<\/p>\n

My situation is that I have the database userid and password (for live server) which does not have superuser privileges. So here we go.<\/p>\n

Export Database<\/span><\/strong><\/p>\n

My personal experience in using phpmyadmin for exporting large database is not good, so I suggest using php scripts available on internet to import large sql files eg. phpminiadmin or any other. Also you can write your own script. Simply put the script on your testing server via ftp and browse the url of this script, put database credentials and export it<\/p>\n

Other options are using command line or SSH client for Windows which is explained below<\/p>\n

Let us assume username is abc <\/em>and password is abc123<\/em>, the testing server from which you have to export database is example.com <\/em>and database name is test_db1,<\/em> therefore command to import it from your local Windows machine is.<\/p>\n

mysqldump \u2013u abc \u2013p \u2013h example.com test_db1 >> c:\\dbsqlfile.sql <\/span><\/p>\n

Enter the password (in this case abc123<\/em>); of course you can use any destination path instead of c:\\dbsqlfile.sql<\/em> on your local window machine.<\/p>\n

If you are able to SSH to your testing server (example.com<\/em>) then put the following command in SSH client for window like putty or winscp<\/p>\n

mysqldump \u2013u abc \u2013p test_db > httpdocs\/dbsqlfile.sql<\/span><\/p>\n

Enter password and remember that this command will save the sql file on the testing server (example.com<\/em>) to the defined path in above command. \u00a0Here you have to be careful while choosing destination path for saving the sql file because the folder may have permission issues. Now using secure file transfer protocol you can download this sql file on your system<\/p>\n

Import Database <\/span><\/strong><\/p>\n

If you use phpmyadmin installed on the live server to import the large sql file chances are you may get \u2018maximum time execution\u2019<\/em> error even if your file is smaller than the maximum allowed size to upload on phpmyadmin (that was the problem with me). So here is what you should do (some of you would that think changing the maximum execution time setting will help but in my case it didn\u2019t).<\/p>\n

Suppose your live server is liveexample.com,<\/em> database name is test_db2<\/em> with username xyz<\/em> and password xyz123<\/em>. Remember this user has only permission to access database test_db2<\/em>.<\/p>\n

On your local Windows machine open command line tool and enter the following command<\/p>\n

mysql \u2013u xyz \u2013p \u2013h liveeaxample.com test_db2 < c:\\dbsqlfile.sql<\/span><\/p>\n

Enter your password (xyz123<\/em>). Remember to use the correct source path.<\/p>\n

If you have uploaded your database sql file on the live server then you can use the free SFTP, FTP and SCP client for Windows called Putty or WinSCP and enter the following command<\/p>\n

mysql -u xyz \u2013p test_db2 < httpdocs\/dbsqlfile.file<\/span><\/p>\n

Enter your password (xyz123<\/em>), again remember the correct path of the database sql file (it is the path where you have uploaded sql file on live server). You can also use source<\/em> command after login in mysql for the same.<\/p>\n

One important thing I would like to mention that the process of exporting database from testing server on your local computer and then uploading it on live server for importing it on the database, can be done in many other efficient way which is a lot faster than this process, if you know tar<\/em>, scp<\/em> or rsync<\/em>\u00a0 unix commands to transfer file over a network you can directly transfer your large sql file from one server to another \u00a0without any problem. I myself used tar<\/em> command to transfer large file (though not in this case). Even if you are not very experienced user you can Google the command and shoot it on any SSH client program like putty<\/p>\n

Coming back on main topic, there are chances that after executing the above commands for importing the database you may get Error \u201cGot a packet bigger than ‘max_allowed_packet’ bytes<\/em>\u201d. This is the major problem which I faced during all these steps and it also encouraged me to write this blog.<\/p>\n

For this you have to change the max_allowed_packet<\/em> value for your local system mysql daemon as well as live server mysql daemon.<\/p>\n

If you have superuser privileges for your database user xyz<\/em> than you can do the steps below without any error.<\/p>\n

SSH to the live server, login to mysql console connected to the live server and run these commands<\/p>\n

set global net_buffer_length=1000000;<\/span><\/p>\n

set global max_allowed_packet=1000000000;<\/span><\/p>\n

Try to use large value for packet size.<\/p>\n

To check if they’re set, use<\/p>\n

select @@max_allowed_packet;<\/span><\/p>\n

Now open another SSH terminal and shoot the following command<\/p>\n

mysql –max_allowed_packet=100M -u xyz -p test_db2 \u00a0< httpdocs\/dbsqlfile.file<\/span><\/p>\n

Enter the password (xyz123) and you are done.<\/p>\n

But as I mentioned above that I had no superuser privilege for my database user so running the above mentioned command still failed with error message \u201caccess denied; you need the SUPER privilege for this operation- \u2026..<\/em>\u201d. So what to do now? Either you can ask the hosting company to give you the password of the root user (which has all the privileges) or take permission to change mysql root password (which will not be allowed in shared hosting unless you have a dedicated server) and do the importing stuff as mentioned above. Another thing you can do is you can ask the hosting company to grant all privileges to database user xyz<\/em>.<\/p>\n

I assume that you have got the permission to reset the root password, so here is the way you can reset the root password (This technique can also be used to reset the root password of mysql server in any other case)<\/p>\n

Log in as root user (remember only root user has all the privileges to execute commands, this root user is not same as the database root user, it is root user of live server) in your live server using SSH client<\/p>\n

Stop the mysql server:<\/p>\n

service mysqld \u00a0stop<\/span>\u00a0 or \/etc\/rc.d\/init.d\/mysql stop<\/span><\/p>\n

start mysql server without password:<\/p>\n

mysqld_safe \u2013skip-grant-tables &<\/span><\/p>\n

connect to mysql server:<\/p>\n

mysql \u2013u root<\/span><\/p>\n

Now start the process of setting password for root user:<\/p>\n

mysql> use mysql;<\/span><\/p>\n

Database called mysql<\/em> has a table called user<\/em> which contains all the users information including root<\/em> user, sometime there is no root<\/em> user instead you can find user with name admin<\/em> which has all the privileges, so in this case you have to reset the password of admin<\/em>. To list all the users in user <\/em>table you can use following command.<\/p>\n

mysql> select User form user;<\/span><\/p>\n

I myself found admin instead of root. Now use update command to reset the password of the root<\/em> user<\/p>\n

mysql> update user set password=PASSWORD(“NEW PASSWORD”) where User=’root’;<\/span><\/p>\n

where NEW PASSWORD<\/em> is your new password which you want to set for the root user (remember you may have to change root<\/em> with admin<\/em> in above command if you found user admin <\/em>instead of root<\/em>). Now to retain all the privileges run<\/p>\n

mysql> flush privileges;<\/span><\/p>\n

mysql> quit;<\/span><\/p>\n

After quitting from mysql client again stop msyql server:<\/p>\n

service mysqld stop<\/span>\u00a0 or \/etc\/rc.d\/init.d\/mysql stop<\/span><\/p>\n

Again start mysql server and try your new password<\/p>\n

service mysql start<\/span> or\u00a0 \/etc\/rc.d\/init.d\/mysql start<\/span><\/p>\n

mysql \u2013u root \u2013p or mysql \u2013u admin \u2013p<\/span><\/p>\n

Enter your new password and you will login in mysql. You can set max_allowed_packet and net_buffer_length and able to import database using large sql file as mentioned above without any error.<\/p>\n

This is all you need to export and import a large database, hope you enjoyed the blog.<\/p>\n

I would like to hear your suggestions and comments<\/p>\n","protected":false},"excerpt":{"rendered":"

During my recent project, i was in a situation where I had to move a large mysql database from a testing server to live server. […]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":1090,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,279,278,10,11,1],"tags":[281,147,280,186,200],"class_list":["post-1084","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apachephp","category-mysql-programming","category-mysql-2","category-programming","category-tutorials","category-uncategorized","tag-export","tag-import","tag-large-database","tag-mysql","tag-php"],"_links":{"self":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/1084","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/comments?post=1084"}],"version-history":[{"count":14,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/1084\/revisions"}],"predecessor-version":[{"id":1101,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/1084\/revisions\/1101"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/media\/1090"}],"wp:attachment":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/media?parent=1084"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/categories?post=1084"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/tags?post=1084"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}