Pack up the game resources to reduce loading time

Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
Developers, with all due respect, i'm going to say this loud:
No other game has raped my HDD so hard as NS2. Ever.
Seriously, with every map change it makes such horrible noises like it's going to disintegrate into shrapnel any second.

The reason why other games aren't putting so much pressure to HDD is that their assets are packed into bigger files.
Loading over nine thousand small files scattered around the disk could even be worse than reading a huge file fragmented in small chunks. (because the OS need to check access rights, write down time of last access etc)

As the solution to this i suggest you to bundle assets into big files (packs). This way they can be loaded without constant HDD's heads jerking back and forth. It could improve load times and more importantly won't chew off HDD lifetime so much.

Modded files could be loaded after (or even instead of) the core packs. Or a list of modded files could be built to make the loading script skip chunks of a big file that will not be needed.

P.S. If you're going to make me choose between my HDD and NS2 i'll probably choose HDD.
P.P.S. "Buy SSD" is not a valid solution. Like i said, other games aren't doing such things.

Comments

  • cooliticcoolitic Right behind you Join Date: 2013-04-02 Member: 184609Members
    edited November 2014
    I dont experience this problem, and neither do most people with HDD's.

    You probably need to replace your HDD.
  • Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
    edited November 2014
    My HDD was quite loud from the beginning, it's not a malfunction. The heads' motors are making a distinct noise so i can hear their every movement quite clearly. But in other games they make only few repositions while in NS2 it's constant and very significant moves. Your HDD probably makes the same moves but quietly.
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    do you not play battlefield 3 or 4?
  • Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
    edited November 2014
    IronHorse wrote: »
    do you not play battlefield 3 or 4?
    Tested BF3. Loading took about 30 seconds without significant heads' jerking.
    People complaining about 2 minutes loading times of BF4 with 5400RPM HDD. It's nothing compared to 5+ minutes of motorboat sounds that NS2 makes every map change with my HDD as a speaker.

    So i'd like to see archives thing implemented soon.
  • Kouji_SanKouji_San Sr. Hινε Uρкεερεг - EUPT Deputy The Netherlands Join Date: 2003-05-13 Member: 16271Members, NS2 Playtester, Squad Five Blue
    How much memory does your rig have?
  • Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
    AMD Phenom II X4 2.6Ggz
    4Gb DDR3 RAM
    AMD HD5770 1024mb GDDR5
    Seagate ST3500630AS 7200 rpm 16 Mb cache SATA 3Gb/s

    Unloadng everything RAM-consuming and running NS2 on low settings in hopes to reduce loading load (ha-ha)

  • develdevel Join Date: 2014-09-13 Member: 198444Members
    edited November 2014
    Van_Tuz wrote: »
    Loading over nine thousand small files scattered around the disk could even be worse than reading a huge file fragmented in small chunks. (because the OS need to check access rights, write down time of last access etc)

    So don't use crappy OSes.
    (and buy SSD anyways, it is great)
  • NordicNordic Long term camping in Kodiak Join Date: 2012-05-13 Member: 151995Members, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow
    devel wrote: »
    Van_Tuz wrote: »
    Loading over nine thousand small files scattered around the disk could even be worse than reading a huge file fragmented in small chunks. (because the OS need to check access rights, write down time of last access etc)

    So don't use crappy OSes.
    (and buy SSD anyways, it is great)
    That is not really helpful, because linux is not real stable for ns2 right now. I do agree ssd's are incredibly nice and I would never go back to an HDD for my os and games. OP already stated this was not an option for him though.

  • Kouji_SanKouji_San Sr. Hινε Uρкεερεг - EUPT Deputy The Netherlands Join Date: 2003-05-13 Member: 16271Members, NS2 Playtester, Squad Five Blue
    Running some diagnostics programs (MemTest86 and HDtune) couldn't hurt right. I've had a faulty HDD, which performed well for video encoding, but not for some games loading, thing was it had bad sectors and had to compensate for more than usual write errors. Seems quite similar to the varied performance you have :(
  • cooliticcoolitic Right behind you Join Date: 2013-04-02 Member: 184609Members
    For gaming it's recommended to have 8 GB of RAM.

    Just saying.
  • NordicNordic Long term camping in Kodiak Join Date: 2012-05-13 Member: 151995Members, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow
    coolitic wrote: »
    For gaming it's recommended to have 8 GB of RAM.

    Just saying.

    Meh. 4 is plenty for ns2.
  • ObraxisObraxis Subnautica Animator & Generalist, NS2 Person Join Date: 2004-07-24 Member: 30071Super Administrators, Forum Admins, NS1 Playtester, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Silver, WC 2013 - Supporter, Subnautica Developer, Pistachionauts
    A good and full harddrive defragment will help NS2, so your HDD will have to do less work.

    https://www.piriform.com/defraggler is the best and fastest I've found. Built-in windows one is slow.
  • Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
    Nordic wrote: »
    devel wrote: »
    So don't use crappy OSes.
    (and buy SSD anyways, it is great)
    That is not really helpful, because linux is not real stable for ns2 right now. I do agree ssd's are incredibly nice and I would never go back to an HDD for my os and games. OP already stated this was not an option for him though.
    SSD is an option for me but i don't want to make it easy for the developers. You know, it's easy to say "we're too lazy to optimise our game so just buy better hardware" (no offence) This is a bad development practice and should not be excused. For the sake of the game and games in general. Or else we just won't have SSD space for every game that wants to live there.
    Why a game needs to load thousands of tiny files when it can load a few big file packs faster and easier? This
    is not an OS' fault.



    I've ran HDtune error scan. No bad sectors. Reallocated sector count is 72. Far from critical i think.
    More importantly, during the scan the HDD was completely quiet except for a few minor seeking noises when other things asked for HDD time.
    Defraggler shows that no NS2 file is fragmented.
    I think that should be enough to show that my HDD is fine. I don't want to prove that any further.



    If someone interested, i've put the microphone on top of HDD and recorded the loading process.
    Caution: Extreme violence. Your HDD/SSD should be at least 1 year old to proceed.
    https://www.dropbox.com/s/99x1sezs32fpy4h/NS2 loading.mp3?dl=0
    It gets good at 2:10 mark when "pre-caching" kicked in.
    The files are just scattered around the disc so HDD needs to reposition the heads to read every single one. Don't forget that extensive seeking not just increases loading times and power consumption but also eats HDD mechanics' lifetime.

    BTW, despite the 7 min loading when i first time saw a skulk, the game decided to hang for a second to load the pretty textures. The skulk surely liked the taste of fresh marine legs.



  • develdevel Join Date: 2014-09-13 Member: 198444Members
    edited November 2014
    Nordic wrote: »
    devel wrote: »
    Van_Tuz wrote: »
    Loading over nine thousand small files scattered around the disk could even be worse than reading a huge file fragmented in small chunks. (because the OS need to check access rights, write down time of last access etc)

    So don't use crappy OSes.
    (and buy SSD anyways, it is great)
    That is not really helpful, because linux is not real stable for ns2 right now. I do agree ssd's are incredibly nice and I would never go back to an HDD for my os and games. OP already stated this was not an option for him though.

    Just saying that he could write to Microsoft instead of Unknown Worlds Entertainment.
    Because, I quote,
    Van_Tuz wrote: »
    (no offence) This is a bad development practice and should not be excused.

    Well, if you don't understand the cause of the problem, you have a right to write the game developer something like this: "game load is slow; my HDD is bleeding".

    But if you know about that situation with thousands of files and that you have to defragment them, then you can now leave game developer alone and deal with another problem: reading thousands of abstract files from the disc.

    However, you still choose to write to the game developer. Probably, because it is smaller.
    Yes, bigger companies are too big to bother. So, they can follow "bad development practices" safely, because smaller ones will workaround these bugs under users' pressure.

    P.S. Do what you do (cos u aint getting ... done other way for now). But keep in mind a deeper background, it's a step forward already.
  • Van_TuzVan_Tuz Join Date: 2014-11-04 Member: 199412Members
    edited November 2014
    If you're going to make this personal, Devel...
    First,
    devel wrote: »
    Just saying that he could write to Microsoft instead of Unknown Worlds Entertainment.
    Do Linux users have <30 seconds loading times from a HDD? If not then it's not a OS' fault and no need to make an OS holywar here.

    Second:
    Do you know the difference between file fragmentation and file scattering?
    I have none of NS2 files fragmented. But they're scattered. That way a lot of time is wasted seeking back and forth. Defragmentation tools are not as effective at descattering as at defragmentation.

    Third:
    Note that i'm not spewing hate here. I'm suggesting a possible solution: Compose the small files into big ones. Reading one continuous file is faster than reading a hundred of small ones.

  • _INTER__INTER_ Join Date: 2009-08-08 Member: 68392Members, NS2 Playtester, Reinforced - Shadow
    My HDD is also still behaving crazy when loading NS2. Monitor the loading with "resource monitor" (cmd > resmon). HD Tab, scroll down and arrange the table headers that you see them all. Could you post a couple of screenshots of that?
  • develdevel Join Date: 2014-09-13 Member: 198444Members
    Apparently, you know the difference between file fragmentation and file scattering. And are able to find descattering software if needed.

    I'm explaining why you are suggesting this or that solution.
  • IeptBarakatIeptBarakat The most difficult name to speak ingame. Join Date: 2009-07-10 Member: 68107Members, Constellation, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Diamond, Reinforced - Shadow
    I'm backing up Van_Tuz here in saying the loading is terrible and the performance is generally a mess.

    People also really need to stop fanboy'ing when it comes to these kinds of complaints, it isn't helping anyone when you're defending the software issues by telling people they should throw hardware at a software problem. Otherwise we should just up the system requirements to have the minimal systems requirements include a GTX 780 Ti, a SSD, and 8-16Gb of ram because some of us can barely get the minimum enjoyment of the game with specs greater than the current recommended.
  • cooliticcoolitic Right behind you Join Date: 2013-04-02 Member: 184609Members
    Van_Tuz wrote: »
    Third:
    Note that i'm not spewing hate here. I'm suggesting a possible solution: Compose the small files into big ones. Reading one continuous file is faster than reading a hundred of small ones.

    @GhoulofGSG9 just said they were working on something like this. No need to extend it further.
  • develdevel Join Date: 2014-09-13 Member: 198444Members
    @Van_Tuz, ok, I'm sorry that I offended you.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    Van_Tuz wrote: »

    Reallocated sector count is 72.
    ... my HDD is fine.

    ... also eats HDD mechanics' lifetime.

    ;))
  • ObraxisObraxis Subnautica Animator & Generalist, NS2 Person Join Date: 2004-07-24 Member: 30071Super Administrators, Forum Admins, NS1 Playtester, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Silver, WC 2013 - Supporter, Subnautica Developer, Pistachionauts
    We are working on ways to help with your problem, but they will not be a quick thing and will take time to develop.
Sign In or Register to comment.