We Rule status update from Neil Young

Posted on: 3/21/2010


Dear Everybody,

We are tracking down a few at-scale issues that we hoped to have fixed soon. I thought you might appreciate some transparency as we address these problems. Here’s what we’re working on right now to ensure We Rule operates as designed…

The first, is a database deadlock, which we believe is most likely causing the connection failures that you are seeing - as the deadlocks occur, that locked database becomes unresponsive and the application processing queues get backed up. The impact is ever increasing response times, which in turn impacts things like your crops hanging at 99% and timing out vs. getting harvested, until eventually those queues become full and you cannot connect. We can manually reset these, but the reality is that we’re now seeing hundreds of thousands of DB requests every minute and so as demand grows we really must find & fix this issue.

The second issue, is related to the number of friends & followers you have. We Rule requests friends from Plus+ on boot and when that list changes, the game then uses this information to populate the social map at the beginning of the game and to identify users who have pending jobs in your kingdom. The game is requesting both friends and followers, and given that the average number of friends per user is 11, and the number of followers per user is much, much higher this is creating a lot of additional load. We have a fix for this issue in our staging environment and will be testing it before pushing it live.

The final (for now) issue is the user reports of inconsistency in how your kingdom is behaving - you may put down a building, only to find that it’s not there the next time you come back, or plant a crop to see it mature almost immediately. These are issues that we have seen before in our Canada test and believed that we had addressed before going global - they are at the nexus of server load and caching. If these are indeed the same issues that we saw in Canada, then the consolation for you is that the state that you have on the server is actually the true state of the game and that these issues are more confusing than they are a permanent problem with your kingdom. We hope that the net result of addressing the issues above will be that your kingdom is restored to it’s accurate state. We monitor your tweets & posts very closely and will carefully watch the progress here and if the issues persist we will certainly address and correct them.

We do not take service issues lightly and please rest assured that everyone who can address these issues at ngmoco & newtoy are working around the clock to get the game in the right state.

Thank you for your patience and patronage, we’ll keep you updated on progress through our @ngmoco & @werulegame Twitter accounts.

—- Neil Young, Founder & CEO

posted : Sunday, March 21st, 2010