Home     About     Command List     Rules     Forums     Donate     Maps     Contact Us

 

Understanding WHY big worldedit changes crash the server.

<<

irokin

User avatar

Posts: 59

Joined: 28 Jun 2011, 3:47 pm

Post 13 Jul 2011, 1:56 pm

Understanding WHY big worldedit changes crash the server.

I think its important some of you newbie (I mean this in a friendly way) admins understand why your actions have the affect they do on the server.

Why can I use worldedit in single player to place 90,000 blocks of TNT, set them off and not have it crash completely? It comes down to logging. Every block placed or removed in world currently is logged. These actions, the user, coordinates and so forth are stored in a MySQL database.

When you make large changes the anti-grief logging starts feeding this data, query by query to the database. Now, a single large SQL query can execute pretty quickly. When you feed the database lots of small discrete queries performance degrades significantly. I seriously doubt the logging mod has been written to take advantage of the single query performance so that when many blocks are changed at once, this is fed as one huge single query or split into a few large queries.

So why does this matter? Because roly has the logging table setup as MyISAM this means that every write, update, delete (and some others) query fed to the database causes the ENTIRE table to be 'locked'. What does this mean? Locking (in this context) the table means that no other query can WRITE or READ from the table. This means that if you change 100,000 blocks, you create a queue 100,000 queries long. That means the table is going to be LOCKED, WRITTEN and UNLOCKED an absolute minimum of 100,000 times. As you can imagine this is going to take some time. Here's some stats from one of my databases (not minecraft related):

Queries:
Total: 291 M
ø per hour: 41.26 k
ø per minute: 687.64
ø per second: 11.46

Now that runs fine and fast for me for a few reasons. One, most of my database queries are reads not writes. Two, the server is running a RAID 10 array of WD RE3 drives, blisteringly fast. Third, I've done considerable optimisation to make the database fast for my query profiles. On tables where I do most of my writes and updates I use InnoDB which only locks the row of the database that it needs to, leaving the rest of the table open for reads and writes. This means everything doesn't grind to a halt just because there's a few writes pending but there are other downsides (MyISAM has better performance in some respects along with full text indexing).

I suspect where the crashing or just lag comes into play is that the database isn't responding to READ requests in a timely manner because its busy trying to write the queue to the disk. The write queue processes slow because the hard disks now can't keep up with the write requests. Now you start to cause other problems. Because the disks cant keep up, the minecraft server starts having troubles writing its own database of block values. Also, the process threads that are trying to write or read data to either of the databases become essentially locked. They're waiting for data that's going to take a while to get to them which means if all the threads get locked the servers no longer sending data to the players and thus you get "End of Stream" or just java socket errors.

I don't mean to be presumptuous in this post and tell you all how to run the server. I'm not an admin, that's not my role. I felt it necessary to express the concerns some other players have discussed with me as a fellow player. I thought the best way to address this might be attempting to explain what happens behind the scenes when these incidents happen. You should take pride in helping to maintain a stable, lag free server. When you have Op's powers, you're no longer a luser and you need to treat these powers with the respect they deserve. Double check your commands before you commit them. How big was that area I just selected? Could I have mistyped the coordinates? That command I just commited didn't seem to do anything? Then you should probably //undo because you can bet your bottom dollar it DID make a change even if you cant see it. If you don't understand world edit commands properly (and I'm not going to profess to be an expert with them either) download single player mods and play around with them offline. It can be a learning curve but don't force others suffer through your mistakes :)

Hopefully this helps in understanding why what you're doing causes crashes and/or lag and think about how you might avoid these problems in future.

tl;dr version: Don't change massive amounts of blocks. Use //size if you're not sure how much you've selected. If you MUST make huge changes, discuss with roly how the best way to go about it would be. This might involve taking the server offline while its quiet, making the changes and then bringing it live again.

Cheers :)
 

Login to remove these ads

<<

irokin

User avatar

Posts: 59

Joined: 28 Jun 2011, 3:47 pm

Post 13 Jul 2011, 3:27 pm

Re: Understanding WHY big worldedit changes crash the server

Oh and to give you idea of how slow sequential queries can be compared to a singular query. I had written a program on my old Celeron D work laptop that was taking about 150 values from a CSV file and writing them to a database. The script was taking about 25 seconds to write that many values, insanely slow. Instead of writing directly to the database I changed it to feed the values into an array and then insert all of the data in one query. Execution time went from 25 seconds to sub 1 second. Massive performance gain based largely on the disk performance. So extrapolate that to 100,000 and you're potentially looking at about 4 HOURS to write all of that data on that laptop. Faster on modern machines of course but its still going to take a while. The drive heads are going to take about ~10ms per seek so there's at least 1000 seconds (16 minutes) already.
<<

Icersau

Game Master

Posts: 97

Joined: 26 Jun 2011, 8:00 pm

Post 13 Jul 2011, 3:58 pm

Re: Understanding WHY big worldedit changes crash the server

You lost me at "I"....... Nerd :p
<<

crusty_krow

User avatar

Posts: 373

Joined: 23 Jun 2011, 9:30 pm

Location: Launceston, Tasmania

Post 13 Jul 2011, 4:08 pm

Re: Understanding WHY big worldedit changes crash the server

BRAIN OVERLOADED *goes to blow up irokin's house*













loljks
Image Image
<<

rolyd90

User avatar

Game Master

Posts: 1241

Joined: 13 May 2011, 7:15 pm

Location: Adelaide, South Australia

Post 13 Jul 2011, 6:02 pm

Re: Understanding WHY big worldedit changes crash the server

i have to say irokin knows what hes talking about :)
<<

dale00474

User avatar

Game Master

Posts: 1737

Joined: 30 Jun 2011, 11:28 pm

Location: Adelaide, SA

Post 13 Jul 2011, 7:53 pm

Re: Understanding WHY big worldedit changes crash the server

Do keep in mind what happened last night was a case of screwing around gone wrong. Not sure about some of the guys but I actually have a good deal of development experience with a few MMO's. Needless to say outside of logs their is a good deal of physics that must be rendered and computed with everything done in game as well. But you have made some good points that play into actions. I found a gui utility that I posted for the admins for world edit which sjould minimize incidents. But what happened last night was mearly an accident. Dawzie and I thought had a talk today about world changes and their effects on the server. I would love to know if anyone knows what engine the game uses?
-Dale :moderator:In-game admin/GM

Refuge: n. That which shelters or protects from danger or from distress or calamity; a stronghold which protects by its strength or a sanctuary which secures safety by its sacredness; a place inaccessible to an enemy.
<<

dale00474

User avatar

Game Master

Posts: 1737

Joined: 30 Jun 2011, 11:28 pm

Location: Adelaide, SA

Post 13 Jul 2011, 7:58 pm

Re: Understanding WHY big worldedit changes crash the server

By block values I assume you are talking on physics and relative world dynamics?

I'd be interested in talking with you on these things in steam or TS sometime. =)

Ok...by MMO one was WOW...there I said it...lol. But it was in the time of burning crusade... =D
-Dale :moderator:In-game admin/GM

Refuge: n. That which shelters or protects from danger or from distress or calamity; a stronghold which protects by its strength or a sanctuary which secures safety by its sacredness; a place inaccessible to an enemy.

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 2 guests


Powered by phpBB
Copyright 2016 by Roland Dadd
cron