Game Coding
Vanesyra
Join Date: 2013-09-02 Member: 187705Members
Join Date: 2013-09-02 Member: 187705Members
For the last days I spend many hours in looking into the lua code from NS2. First I must admit that I did not really used lua in a bigger environment on my own before, or have any into the deep experience with it's special lua specific performance topics. So some stuff might be wrong, that I am going to say here.
The first thing I have ask myself is why lua and why not javascript? There is no good IDE for lua, but there are many for javascript. Sure you have written Decoda, which can be considered to be the best IDE for lua, but still in compare to it, it is way behind stuff like visual studio express web developer. And on the performance topic, a javascript v8 engine should perform at least at the same speed, luajit does. The only thing that is a plus for lua is that at the time you started developing, lua might had the better performance. But looking at it on the current day, lua is a language very less people can use (with a "special" syntax), very less tools use and is very limited. While javascript can be considered as the language that most people understand, and it's performance aspects are very well known. So for a modding friendly environment, javascript would be the way to go.
Next thing is when looking at your codeing style, you did not do very well in using documentation on it. Most functions have no comments at all. New lines are added more or less random into the code, and do not follow a real pattern. Making stuff harder to read and to understand then it could be. However the naming of variables and function itself is done very well.
If we look at the way you use function calls, I think you spend many cpu time in unneeded function calls. For example the "Marine" and it's "GetActiveWeapon()" this is a painful function call chain, done all over again. And it's not needed. If you switch weapon on the "Marine" store the current weapon in "Marine.activeWeapon" and done. In so many places you simply recalculate the same stuff over and over again, this sums up and makes stuff slow.
Very much the same thing is on some calculations. For example "Marine:GetIsStunAllowed()". Instead of setting a "Marine.timeStunImmunity" on the "Marine:OnStun()" to "Shared.GetTime() + kDisruptMarineTimeout", you do that calculation over and over again. This is getting even more evil because on ever "Marine:GetIsStunAllowed()" you call "Shared.GetTime()" to and "self:GetIsVortexed()". Just count through the amount of function calls you do in that case when one Onus Stomp is parsed onto 3 Marines for example. You really should do that with only ONE "Shared.GetTime()" call for example. And sure if you only focus on this single function, the impact on overall performance is less. But if you sum it up in all functions all over the code, it is getting evil. Also make more use of function parameters instead of calling function all over again. For example use "Marine:GetIsStunAllowed(time)" instead of "Marine:GetIsStunAllowed()" and calling the time with "Shared.GetTime()" in every single Marine "Object".
What I don't know about lua is if lua makes a difference between float and integer numbers. If not, ignore this. Because calculating "1 + 1.0" is slower then "1 + 1" and "1.0 + 1.0" in some other languages. This is because casting from int to float or the other way needs to be done. You seem to use this randomly for example "Marine.kWalkMaxSpeed = 5" and "Marine.kRunMaxSpeed = 6.0". Also not so nice is using "Marine.kWeldedEffectsInterval = .5" and on other parts "Marine.kWalkBackwardSpeedScalar = 0.4". Take the time and write down the "0"
And then for my opinion you have gone to far with your attempt to make a mod friendly environment. You have basically everything on the lua layer, so in a slow performance environment. The really basic stuff, networking, movement and updates like this should run in native code and should not be part of the mod environment. If you move so far away from the game with your mod, you should use "Unity3D" and do your own game! Think about moving the most used functions (the once with most cpu runtime spend) from lua to native code for a better performance.
Over all I must say that you have one big design error in your mod / lua system. To explain this lets take a look at google chrome webbrowser and the render / javascript engine there. If you have a huge html document open, the html elements (compare them with Players in NS2, and Map elements) exist in the native code environment. They are not a part of the sandbox where javascript runs in. If you call "document.getElement...()" the native code Element is pushed into the sandbox, with function to overwrite and alter it. But the element itself, still is within the native code NOT the sandbox. This is the way they keep performance high and that is one of the reasons NS2 is eating up so much performance here. The much more performance way would be if you code stuff within native language and then let the mod makers overwrite the functions with lua (or better javascript ^^). A fake code example would be: the native code: "Marine:OnCreate() { the native code here for it }" and on the lua / javascript side "Overwrite.Marine.OnCreate = function ( my alternative code here }". The mod could still do everything then, but if it does not overwrite a function the native code is used and at that point the elements do not exist in the scripting sandbox, they exist in the fast native code environment.
To sum this up, many of the performance problems you encounter are house made by yourself. I know you can't just say oh yes lets switch to javascript from today to tomorrow and lets write function in native code and add the ability to overwrite them then. This is nothing to be done over night and on the fast track. But you really should think about what you can optimize on that topic, and how many time / money you can spend on that.
Do not really know if I hit the correct board for that feedback, just wanted to share my opinion on that. Anyway it's still a great game and well done!
The first thing I have ask myself is why lua and why not javascript? There is no good IDE for lua, but there are many for javascript. Sure you have written Decoda, which can be considered to be the best IDE for lua, but still in compare to it, it is way behind stuff like visual studio express web developer. And on the performance topic, a javascript v8 engine should perform at least at the same speed, luajit does. The only thing that is a plus for lua is that at the time you started developing, lua might had the better performance. But looking at it on the current day, lua is a language very less people can use (with a "special" syntax), very less tools use and is very limited. While javascript can be considered as the language that most people understand, and it's performance aspects are very well known. So for a modding friendly environment, javascript would be the way to go.
Next thing is when looking at your codeing style, you did not do very well in using documentation on it. Most functions have no comments at all. New lines are added more or less random into the code, and do not follow a real pattern. Making stuff harder to read and to understand then it could be. However the naming of variables and function itself is done very well.
If we look at the way you use function calls, I think you spend many cpu time in unneeded function calls. For example the "Marine" and it's "GetActiveWeapon()" this is a painful function call chain, done all over again. And it's not needed. If you switch weapon on the "Marine" store the current weapon in "Marine.activeWeapon" and done. In so many places you simply recalculate the same stuff over and over again, this sums up and makes stuff slow.
Very much the same thing is on some calculations. For example "Marine:GetIsStunAllowed()". Instead of setting a "Marine.timeStunImmunity" on the "Marine:OnStun()" to "Shared.GetTime() + kDisruptMarineTimeout", you do that calculation over and over again. This is getting even more evil because on ever "Marine:GetIsStunAllowed()" you call "Shared.GetTime()" to and "self:GetIsVortexed()". Just count through the amount of function calls you do in that case when one Onus Stomp is parsed onto 3 Marines for example. You really should do that with only ONE "Shared.GetTime()" call for example. And sure if you only focus on this single function, the impact on overall performance is less. But if you sum it up in all functions all over the code, it is getting evil. Also make more use of function parameters instead of calling function all over again. For example use "Marine:GetIsStunAllowed(time)" instead of "Marine:GetIsStunAllowed()" and calling the time with "Shared.GetTime()" in every single Marine "Object".
What I don't know about lua is if lua makes a difference between float and integer numbers. If not, ignore this. Because calculating "1 + 1.0" is slower then "1 + 1" and "1.0 + 1.0" in some other languages. This is because casting from int to float or the other way needs to be done. You seem to use this randomly for example "Marine.kWalkMaxSpeed = 5" and "Marine.kRunMaxSpeed = 6.0". Also not so nice is using "Marine.kWeldedEffectsInterval = .5" and on other parts "Marine.kWalkBackwardSpeedScalar = 0.4". Take the time and write down the "0"
And then for my opinion you have gone to far with your attempt to make a mod friendly environment. You have basically everything on the lua layer, so in a slow performance environment. The really basic stuff, networking, movement and updates like this should run in native code and should not be part of the mod environment. If you move so far away from the game with your mod, you should use "Unity3D" and do your own game! Think about moving the most used functions (the once with most cpu runtime spend) from lua to native code for a better performance.
Over all I must say that you have one big design error in your mod / lua system. To explain this lets take a look at google chrome webbrowser and the render / javascript engine there. If you have a huge html document open, the html elements (compare them with Players in NS2, and Map elements) exist in the native code environment. They are not a part of the sandbox where javascript runs in. If you call "document.getElement...()" the native code Element is pushed into the sandbox, with function to overwrite and alter it. But the element itself, still is within the native code NOT the sandbox. This is the way they keep performance high and that is one of the reasons NS2 is eating up so much performance here. The much more performance way would be if you code stuff within native language and then let the mod makers overwrite the functions with lua (or better javascript ^^). A fake code example would be: the native code: "Marine:OnCreate() { the native code here for it }" and on the lua / javascript side "Overwrite.Marine.OnCreate = function ( my alternative code here }". The mod could still do everything then, but if it does not overwrite a function the native code is used and at that point the elements do not exist in the scripting sandbox, they exist in the fast native code environment.
To sum this up, many of the performance problems you encounter are house made by yourself. I know you can't just say oh yes lets switch to javascript from today to tomorrow and lets write function in native code and add the ability to overwrite them then. This is nothing to be done over night and on the fast track. But you really should think about what you can optimize on that topic, and how many time / money you can spend on that.
Do not really know if I hit the correct board for that feedback, just wanted to share my opinion on that. Anyway it's still a great game and well done!
Comments
Yes most of the performance issues are "house made by themselves" but I'd say that's the case for most programs. It can ALWAYS be better (in a realistic world where people aren't robots).
It is certainly more pro-active to have something there to help the team, rather than just sitting there and telling them what they got wrong..
You should read some stuff about javascript and especially the javascript v8 engine to get a impression on how powerful this stuff is now a day. And it's all, but not stupid to use this as a scripting language within a game. One of the big cross platform gaming engines called unity 3d is using javascript for that topic too. If you wanna read more about this visit: http://docs.unity3d.com/Documentation/ScriptReference/index.html Think javascript is the language that has most developed within the last few years, especially in performance. With google pushing hard on it.
@Soul_Rider
You may got me wrong on this, it's not about "sitting there and telling". It's about looking what is wrong and giving a feedback on it. It took me quite some hours of reading code to sum up into that impression and give an as detailed as possible feedback on it. I can not fix most things I noticed there from "outside" and none without them knowing what I am doing / telling them here. So I have to start with what's wrong and then we will see if there is a way to fix something or not.
Just to give you a bit off impression here, if you change function calls and parameters as I suggest, you will break nearly all mods. That use one of that functions, so it's no simple "submit code chance and done" stuff. You have to do this with a plan behind your actions and knowing the reasons why to do so.
Now, if you're arguing for coding speed and capabilities python would have been a much better choice. I think it would have been a much better allround choice, not so much for the execution speed (which can be increased by PyPy's JIT), but for it's very powerful syntax and highly extensive standard library. Unlike lua or javascript, it's more of a complete programming language, and less then a scripting engine [although often used as such] which really has advantages. They could have exposed C/C++ engine functions too, with either using python's API directly or something like Boost.Python; in addition, either lua/js will still require a hackjob when it comes to certain features. Like multi-threading/processing (supported in PY - but, better in PyPy or Stackless Python with micro threads and no GIL).
Now they've probably gone with Lua because of the small core, but that's kind of the problem: They extend it so much, that it isn't small anymore and with a more complete language to begin with, they would not have been need to do so and, it would have been implemented in IDEs.
Besides, on the optimization:
You can always make mirco-optimizations to the code, but in the end the question is will it pay off in the end? Optimize a piece of code to run in 10ns instead of 100ns, but it still might make almost no difference despite a 10x speedup, or a massive one, if the piece of code is extensively used, and it's not once 10ns of cpu time, but several thousands of times...
I feel like what the lua thing mostly lacks is the use of multi-threading in lua, there can possibly be many cases where it might benefit from threading, even with the additional overhead; allthough minor, I guess quite a bit of code can just safely run in a threads, especailly if it's not writing, or if the writing only happens at the end after passing several checks; like tracing things, checking collisions, and stuff, before you do anything, and then you just wait for the other threads to finish before writing the data and, while there is overhead, it should run faster on dual+ core systems which is the standard these days and it is a must for the servers, which usually utilize more cores then clients do.
But of course they could do other optimizations as well, such as caching a lot of the more expensive caclucations, as accessing is usually faster then re-cacluatiing (but more memory intensive, but I think we are lacking in the CPU deparment not so much memory (memory leaks in this build excluded)).
Also another hotspot they could be looking at is algorithmic optimizations, possibly there are hidden extensive loops that could be smaller (like O(n) instead of O(n**2) )
If I understand it correctly, they want to make a game that is very good mod able. Where in the best case, everyone can create some cool things. If you look at all programming languages out there, then you will end up with the result, that javascript is probably the most "spoken" language of all of them. Just because people and especially none professional people use it for creating their own little web sites. This is the same group of people that might create a mod for NS2 or other games. They have to learn lua and understand it, same for python and they will probably not do so just for creating or working on a mod.
The main problem if people hear about "javascript" they think back on IE6 javascript performance
However I do not expect them to change the scripting language, I think it would simply cost to much money at this point.
You are absolutely right that you must look on the functions that cost most performance and start optimizing them and not other stuff. That's the way optimization is done! I just wanted to give an example on the "common" coding performance errors I have found that is easy to understand, even if you are a none coding person.
I do not agree on that multi core topic. The stuff done within LUA here is in the end nothing big. It's no big calculation that needs multi threading, in the end its a pure server / client sync like you had it in Half Life 1, and so are the cpu time that is required. So if you leave the game logic with the lua stuff in a single thread you MUST be fine. You can put other stuff in other threads of course! But I really see zero reasons why the lua needs splitting.
And I also agree on you that memory on that topic is no problem (doing some little cache here and there will not cost more then 1 MB ram, in the evil scenario that you could image).
On the topics with algorithmic optimizations and other optimizations on performance critical parts of the code. Maybe it would be a great idea if they post critical code somewhere (maybe even here within the forum). Think there are quite a few people around here that could help with some idea on some of the things. But it's hard to debug this kind of stuff from the outside. How much time is spend in what kind of functions? How deep is the function call chain?
Now ns3 on the other hand...
Now about multithreading. I think it's bad for scripting to use it. Scripting should be simple even for non-professional programmers and multithreading brings concurrency and synchronization which drives me mad. This kind of stuff should be implemented in the engine and there could be a possibility to interact with it from scripts. For example, by submitting tasks to a queue which is then processed by a worker pool or such. BTW, JS is also single-threaded and that's a good feature, actually.
Lua has some benefits over JS. Its VM is simple enough, compact, stable and quite fast already. It's designed for embedding and also has JIT. It's easy to learn.
These benchmarks show that Lua is slower than JS but after all is it an issue? And also as I understand it that's the comparison of vanilla Lua without JIT.
That said, the main reason why JS isn't suited for NS2 is because the game already uses Lua. Replacing it without any groundbreaking reasons is close to impossible. I've found this list of Lua-scripted games which is quite big and has well-known AAA-titles in it. I couldn't find such a list for JS (except browser games which are written entirely in JS, obv.). Maybe there's something behind it.
There is no AAA - Game with JS out yet because most studios are still testing it.
Javascript is not the most spoken language. Unless you mean it in a way that a browser understands it, however, not terms of actual programs using it. You only really find javascript in web development, which is where it originiated and which it is really targeted for. If you want to go by most used language, you'll end up with C, C++ and Java;
Now, statistics, which, depending on the method used, have very varing results, but javascript is never the top#1, and usually it's behind python and php:
http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html
https://sites.google.com/site/pydatalog/pypl/PyPL-PopularitY-of-Programming-Language
http://lang-index.sourceforge.net/
Also it seems you ingored some of my original points. Javascript is MORE a scripting language, just like lua. It's used mainly in web developement, so the resources you'll find will relate to webdevelopment. And, within browsers you do not have many features other languages have, like sockets, threads, file i/o. Again, this would be extra work as it has to be implemented extra.
Even PHP has more real-world possiblities then JS by default. Now, if i was a more useful choice, you could tranfer your skills from real-world programming to scripting/coding within the game and counterwise and existing code should work on both, unless you're using specific features. Like I could use any python library that only uses the python functions, like pymysql without any extra requirements.
Now you're also arguing that JS is easy to learn, that applies to other languages as well, but in fact, I think python is even easier to learn because a) less syntactic clutter [ex. no {} or ; ] and b) more use of english language (ex. if a and b: instead of if (a && b) {} ), plus, as I said it has real-world applicances, you can easily go ahead and create small scripts or whole applications to run on your computer with the python interpreter (and on linux, python interpreter is usually included, because there is a fair share of applications written in it). It only might be a little odd if you are used to c-style languages, but it has advantages. The other stuff applies to python too, it has documentation included, as well as documentation generators (and if you're unhappy with those you can grab an alterantive or write your own [which is not that hard!]). It has free and decent IDEs too, hell, CPython comes with it's own IDE called IDLE (though there are much better IDEs).
^^
Regardless, yes, changing the language would be essentially rewriting NS2 besides the C++ core.
You don't need to have a big calcuation to do threading. Nothing big? The game runs so poorly because it caps out on 1 CPU, on both, client and server, though it is worse on the server.
There isn't even anything to disagree about to honest:
a) it's a fact it caps out on 1 cpu and then limits performance due to that
b) it's a fact that by using code that runs on multiple cores that cap can be relieved*
c) It's a fact that most people have multi-cpu systems (only 3.75% don't http://store.steampowered.com/hwsurvey - for these 3.75% it will run slower, and I take my chances here, these old 1-core cpus can't run NS2 properly in the first place anyway)
d) That servers get the powerhorse CPUs before the client market gets them, as such we have 8-core Xeons and 12-core Operterions. Currently, you can only use high end, recent-generation Intel CPUs with large clock speeds (3.3ghz + ) to get ~16-18 players playable on a non-overclocked CPU. With the others CPU cores being idle for the most part, because NS2 server does almost no multi-threading.
*b) Now keep in mind that improved multicore support won't mean it will use an equal distribution on all cores (like 50-50 on duals or 25-25-25-25 on quads) , it means we'll see more usage on other cores and not just all the usage on mainly one core. Obviously there is only so much code that is suitable for threading on multiple (and worth the overhead), but that can also be small functions or methods.
Eclipse is a topic on it's own
As I said already I see zero need of having multithreading for that scripting task. The thread containing this script should be idle most of the time, it's really not a "huge" calculation that is going on there. Btw they are working on "workers" for javascript that run in their own threads.
I simply do not see the benefits on the lua side. I see JS in a huge advantage over lua here, in all the topics you mentioned. To the games that use lua already, I have played some of the AAA titles in that list, and they all have performance issue exactly within the part of the game that is effected by lua scripts. But anyway as you said, and as I said already, they most likly will not change the scripting language at this point anymore. So more or less we could save our time and stop talking about that topic too :=)
@GhoulofGSG9
I noticed that already, that at the point they started with NS2 lua might be the language with the highest performance.
@Omega_K2
With "most spoken" I am not referring to employees, or lines written in the last months. I am talking about persons that are able to read / write or basically understand a programming language. And if you look on the internet with the millions of web pages out there, that people (none professional persons) write every day, you can consider that javascript is the language most people already got in tough with. You do not see lua, c, c++, python or anything else in your every days life. So probably if you ask all NS2 gamers, which programming language they know (in terms of can do something with it) already you should end up with a huge lead for javascript. That's why I would pick javascript first for something like this.
I am sorry if I left out one of your original points. I just do not see why it should be a problem that javascript is a scripting language and that it is used on web development? Stuff like sockets, threads, IO and so has to be done on your own no matter what script language you use, NS2 with lua has done that too. So what is the point here? And I am sorry I don't got the point on PHP paragraph as well.
My referance to "easy to learn" was aiming on that topic that I think that most people that have programming skills already, also already writen some javascript.
I said that you do not need multithreading for the scripting language. I did not say you do not need it at all. Just talking about the lua part, and there you do not need it because the lua part can run within one cpu without any problems. You simply will not cap out that cpu with stuff like that.
Lua was chosen because it is makes for easy, rapid development...and because it is the scripting engine used by a number of other games. Yes it does have performance issues and lack of multithreading support -- but at some point, UW had to make a decision and they went with Lua. Perhaps JS would have been a better choice (I don't know), but we are using Lua and there isn't much that would cause them to change that now...so we have to make the best of it.
Also, regarding Lua vs JS dilemma. Let's wait until biiiig companies adopt JS for their games and decide after that. Being a pioneer is costly and time consuming and I doubt we all want more sporadical issues here and there because of scripting language choice. Lua is the old and well tested solution. This outweighs many if not all JS' advantages for indie teams.
Not going to wade into the on going javascript vs lua debate. At all.
Clearly, the solution is that everyone should use Fortran
NS2 used to use Source engine, changed to their own engine and that will never change.
Also what is fortran?
program hello print *, "Hello World!" end program helloyeah i go to that location but there isnt any code, the only two things that i seee are 2 xml files and next all the folders of the game but any
source code where is it? i have visual studio 2008 and 2010 but i didnt see any code can someone help?
Isn't that self explaining lua sources are in lua maps in maps models in models and so on
I looked up a nice example on how you optimize v8 code and how fast it is in compare to c++: http://v8-io12.appspot.com/#1
This should give an impression on how fast and powerful javascript with v8 has become. Sure you will not always get the same performance as c++ code, but you stick very close to it when writing a bit optimized javascript code. Anyway as I also said, I don't think that NS2 will switch to javascript, because it simply cost to much money, money they will / should better spend on other topics. Only if it turns out that the real performance bottle neck is lua script engine, I see a possibility that we see a switch. Because then performance can only be fixed with switching the engine to a faster one, which v8 should be. But the answer on that question can only be given from the developers of NS2, so we should stick on the "optimizing the lua code" topic, where we can get performance boosts as well.
Would be really nice to get some lua script performance logs from something like a 30 mins alien and marine game.
P_logall in the console (there's also a server command) will record every single frame and where exactly resources are spent.
You need to install python to view the log though which will be in your hidden appdata/natural selection 2/ folder (the logs can be big if you record for a while) and then just run the "perfanalyzer" in the ns2/util folder to view it.