Map format documentation

MurphyIdiot

Hey everyone,

I got a request to release some information about the map format. So this is some very basic documentation about the format. I hope it helps and if somebody can copy this onto the wiki and any other sites with this type of documentation, that would be helpful.

The first 3 bytes of the file are a magic number which are the characters "LVL".

The next unsigned byte is an integer version number for the map format. The engine supports loading older versions of the format in addition to the current version.

Next, the engine loads chunks until it reaches the end of the file.

There are 4 types of chunks:
Chunk_Object (Id 1)
Chunk_Mesh (Id 2)
Chunk_Layers (Id 3)
Chunk_Groups (Id 5)

In the first pass, it only reads the groups and layers as data from these chunks are needed while loading the other chunks.

A chunk consists of:
Chunk Id (an unsigned 32 bit integer).
Chunk length (an unsigned 32 bit).
The chunk itself.

The groups chunk consists of:
Number of groups (an unsigned 32 bit integer)
A list of groups.

Each group consists of:
A wide string name (length of a string is encoded in the first 4 bytes as an unsigned integer).
A boolean field for visibility (ignored by the engine, only used in the editor, booleans are encoded as 32 bit unsigned integers)
A color (also ignored by the engine, encoded as red, green, blue, and alpha, each field being encoded as a 1 byte unsigned integer)
A group Id (an unsigned 32 bit integer)

The layers chunk consists of:
Number of layers (a signed 32 bit integer)
A list of layers.

Each layer consists of:
A wide string name.
A boolean field for visibility (ignored by the engine)
A color (ignored by the engine)
A layer Id (an unsigned 32 bit integer)

The binary reader then resets itself back to the start of the chunks for the second pass. This time the mesh and object chunks are processed, skipping the other chunks.

A mesh chunk consists of a series of chunks specific to a mesh. These mesh chunks are:
Chunk_Vertices (Id 1)
Chunk_Materials (Id 4)
Chunk_Faces (Id 3)
Chunk_Triangles (Id 5)
Chunk_MappingGroups (Id 7)
Chunk_GeometryGroups (Id 8)
Chunk_FaceLayers (Id 6)

There are 2 passes. The first pass processes every chunk except Chunk_FaceLayers. The second pass only processes Chunk_FaceLayers.

A Chunk_Vertices consists of:
Number of vertices (an unsigned 32 bit integer)
A list of Vec3s (each Vec3 is three 32 bit floats representing x, y, and z)
(Note: Post version 8 of the map format there is also an unsigned 8 bit integer after each Vec3 representing a boolean which indicates that smoothing is enabled for the vertex)

A Chunk_Materials consists of:
Number of materials (an unsigned 32 bit integer)
A list of string material names (these will be referenced as an index into this list of materials by other chunks)

A Chunk_Faces consists of:
Number of faces (an unsigned 32 bit integer)
A list of faces where each face consists of:
An angle (32 bit float)
An offset (Vec2)
A scale (Vec2)
A mapping group Id (unsigned 32 bit integer)
A material Id (unsigned 32 bit integer)
Number of edge loops (unsigned 32 bit integer)
(these are skipped while the game is loading)
The border edge loop
A list of edge loops
A edge loop consists of:
Number of edges (an unsigned 32 bit integer)
A list of edges where each edge consists of:
A boolean indicating if the edge is flipped
The edge index (unsigned 32 bit integer)

A Chunk_Triangles consists of:
Number of "ghost" vertices (unsigned 32 bit integer, these are added to the list of all vertices read in Chunk_Vertices)
A list of "ghost" vertices (each vertex is a Vec3)
Number of smoothed normals (unsigned 32 bit integer)
A list of smoothed normals (each smoothed normal is a Vec3)
Number of faces (unsigned 32 bit integer)
Number of triangles (unsigned 32 bit integer)
A list of faces where each face consists of:
Number of face triangles (unsigned 32 bit integer)
A list of face triangles where each face triangle consists of:
3 vertex indices (each index is an unsigned 32 bit integer)
3 smoothed normal indices (each index is an unsigned 32 bit integer)

A Chunk_MappingGroups consists of:
Number of mapping groups (unsigned 32 bit integer)
A list of mapping groups where each group consists of:
An Id (unsigned 32 bit integer)
An angle (32 bit float)
Scale (Vec2)
Offset (Vec2)
Normal (Vec3)

A Chunk_GeometryGroups consists of:
Number of vertex groups (unsigned 32 bit integer)
A list of vertex groups where each group consists of:
An Id (unsigned 32 bit integer)
Number of vertex indices (each index is an unsigned 32 bit integer, these are ignored by the game while loading)
Number of edge groups (unsigned 32 bit integer)
A list of edge groups where each group consists of:
An Id (unsigned 32 bit integer)
Number of edge indices (each index is an unsigned 32 bit intergers, these are ignored by the game while loading)
Number of face groups (unsigned 32 bit integer)
A list of face groups where each group consists of:
An Id (unsigned 32 bit integer)
The number of faces in this group (unsigned 32 bit integer)
A list of faces where each face consists of:
A face index (unsigned 32 bit integer)

A Chunk_FaceLayers consists of:
Number of face layers (unsigned 32 bit integer)
Format (unsigned 32 bit integer, must be 2)
A list of face layers where each face layer consists of:
A boolean representing that this face has layers (nothing else to read for this face layer if this field is false)
Number of layer bit values (unsigned 32 bit integer)
A list of layer bit values where each value consists of:
A bit mask (unsigned 32 bit integer)

The final chuck to cover is Chunk_Object. Each Chunk_Object represents 1 entity.

A Chunk_Object consists of:
A boolean which is true when this entity includes layer data
The layer format (unsigned 32 bit integer)
Number of layer bit values (unsigned 32 bit integer)
A list of layer bit values where each value consists of:
A bit mask (unsigned 32 bit integer)
The entities' group Id (unsigned 32 bit integer)
The entities' class name (string value)
Until the end of the Chunk_Object chunk, properties are read from the chunk

These entity properties are read from chunks. The chunk Ids are Chunk_Property (1) and Chunk_Property2 (2). Only Chunk_Property2 is used in the newer versions of the map format.

These are the different type of properties:
Type_String = 0 (string value)
Type_Bool = 1 (boolean value)
Type_Real = 2 (float value)
Type_Integer = 3 (integer value)
Type_FileName = 4 (string value)
Type_Color = 5 (float value)
Type_Percentage = 6 (float value)
Type_Angle = 7 (float value)
Type_Time = 8 (float value)
Type_Distance = 9 (float value)
Type_Choice = 10 (special value)

Each chunk starts with an Id (unsigned 32 bit integer) and the chunk length (unsigned 32 bit integer)
Each chunk consists of:
A name (string)
A type (unsigned 32 bit integer)
Number of components (unsigned 32 bit integer)
A boolean representing if the property is animated (animated fields are ignored while the engine is loading the entity)
The property is parsed based on the type of property.

For float types:
A list of components where each component consists of:
Component value (32 bit float)
Any components beyond the fourth are ignored
Next the components are assigned based on the type of property
Type_Real, Type_Percentage, Type_Time, and Type_Distance use only the first component
Type_Angle uses component 1 for roll, component 2 for pitch, and component 3 for yaw
Type_Color uses component 1 for red, component 2 for green, component 3 for blue, and component 4 for alpha (alpha defaults to 1 if there is no component 4)

For string types:
A wide string value (converted to non-wide string value while parsing)

For boolean types:
The boolean value
All other components are ignored

For integer types:
A signed 32 bit integer value
All other components are ignored

For Type_Choice:
The choice value (signed 32 bit integer)
All other components are ignored

That should cover the basics. I probably missed some details and may have even made a mistake or 2 but this should at least give you an idea of how the file format works for NS2 levels.

Deco

This is great!
Thanks for taking the time to write this.

Is it possible to get some information on the model format?

I wrote a reader and writer for the level format in LuaJIT.
It's working now, but during development there were a few off-by-one and floating point issues :P
This is refinery read and then rewritten:


There seems to be some undocumented chunks.

These are at the top-level:
Id 4: A single wide-string containing XML about various rendering and camera information. Editor settings, maybe?
Id 6: Always 68-bytes long. First four bytes are "10 00 00 00". 16*4 is 64, so that may be length? I have no clue to what the 64 integers are, though. Tram: all zero. Docking: first is "00 00 00 00", rest are "ff 00 00 00". Descent is crazy.
Id 7: Pathing settings. Is this in the Chunk_Property2 form?

And this is inside Chunk_Mesh:
Id 2: Binary information. Edges, maybe?

Again, thank you for the documentation!

Tharos

Refinery : post earthquake version

trinity.ns

What a video! Now that should be inspiration for future maps/areas!

Tharos

Can you do this for other maps ?

MurphyIdiot

I didn't bother documenting stuff that isn't read by the game. These other chunks are editor only.

Id 4: Chunk_Viewports
Id 6: Chunk_CustomColors
Id 7: Chunk_EditorSettings

Inside Chunk_Mesh:
Id 2: Chunk_Edges

bp2008

I have this devious desire to make a minecraft -> ns2 map converter now, but considering how rarely people play custom maps in ns2 I'm not sure it would be worth my time

And also the need for a working nav mesh would make things ... tricky ...

MCMLXXXIV

@Deco did you put your source code anywhere? This looks kind of interesting

GISP

Natural Selection 2: Glitchmode
Coming to a store neer you!: Fall 2009

Would be awesome to see a map made to look like a compleatly smached up space station, If buildings could just be played on wonky surfaces and the commander view wasnt hindered, this could make for some awesome map designs

McGlaspie

Can we get a sticky on this?
So it not be lost in the depths.

CloneDeath

Clarification (I just ran into this pitfall, this is to help anyone else who may also do this):

His description of the Chunk_GeometryGroups makes it seem like each list is embedded in the previous, when really the Chunk_GeometryGroups contains all 3 lists. To be clearer:

Chunk_GeometryGroups {
    ID = 8 : uint32
    Length : uint32
    NumberOfVertexGroups : uint32
    [List of VertexGroup]
    NumberOfEdgeGroups : uint32
    [List of EdgeGroup]
    NumberOfFaceGroups : uint32
    [List of FaceGroup]
}

VertexGroup {
    ID : uint32
    numberofindicies : uint32
    [List of uint32 (indices)]
}

EdgeGroup {
    ID : uint32
    numberofindicies : uint32
    [List of uint32 (indices)]
}

FaceGroup{
    ID : uint32
    numberofindicies : uint32
    [List of uint32 (indices)]
}

BeigeAlert

Hey guys! So I've been researching Spark's clipboard format, and I believe I have a complete understanding of it now, so I'm going to post my findings here.

Before I get to all the technical details, some information about the clipboard format that the spark editor uses:
-ONLY geometry and objects (props, entities, lights, reflection probes, etc.) can be copied to the clipboard.
-Layers and groups are NOT copied to the clipboard.
-Editor settings are NOT copied to the clipboard.
-The clipboard format is pretty close to the .level format, but differs in a few areas.
-There are a lot of unused values leftover, I suspect, from repurposing the .level code to output to the clipboard.

Okay, now things are going to get a little bit more technical. The data is divided into two distinct areas: geometry and objects. If you only copied one type or the other, you will only see one big chunk, not two chunks but one just happens to be empty, or something like that. Geometry always comes first, then objects

I'll walk through the format with an example I just copied. It consists of a plane that I first cut arbitrarily (forming a 6-sided square, with a diagonal slash going through roughly the middle). I then inset another square into one of the two 4-sided shapes that was created by the first cut. I then made a square, completely separate from the other geometry, gave it a texture, and used the displacement tool on it. In addition to this geometry, I have two entities: a prop_static gorge model, and a reflection_probe.

Here's what that looks like in the editor:

I copy all of this into the clipboard, and can open this up with Java. The MIME type is "application/spark editor".

EDIT: using code tags to preserve my indentations.

Since we have both geometry AND objects in the clipboard, we start with the geometry chunk.
  4-byte integer that is always 1. [b]01 00 00 00[/b] remember... this is all LITTLE ENDIAN
  4-byte integer length of chunk. [b]E9 09 00 00[/b] in this case.
  2-byte "short" integer for chunk ID. [b]02 00[/b] for geometry chunk, or 01 00 for object chunk.
That last value is where the clipboard differs from the .level format.  In the level format, the chunk IDs were always 4-byte integers, not 2.
Now we're inside the mesh chunk, and stuff should start looking familiar to the .level format.  I'm going to walk through it anyways though.
The mesh chunk itself is divided up into 7 smaller chunks, only 5 of which are actually used.
From what I can tell, the chunks are always written in the following order: materials, vertex, edge, face, face-layers, mapping, geometry-groups.
I haven't tried changing the order around, though I suspect this would not be a very good idea. ;)
First up is the materials chunk
  4-byte integer for the material chunk id, always 4.  04 00 00 00
  4-byte integer for the length of the chunk.  In this case: 59 00 00 00
  4-byte integer for the number of materials.  In this case: 02 00 00 00
  for each material:
    4-byte integer for the number of characters in the name string, in this case 28 00 00 00
    for each character:
      1-byte for each character
	that tells us the name of the first material (material 0) is "materials/biodome/biodome_grass.material"
	we repeat thhe above steps to get the second material, which is "materials/dev/dev_floor_grid.material"
That's it for the materials chunk.
Next up is the vertex chunk.
  4-byte integer for the vertex chunk id, always 1.  01 00 00 00
  4-byte integer for the length of the chunk.  In this case: CB 01 00 00
  4-byte integer for the number of vertices.  In this case: 23 00 00 00
  for each vertex:
    float value (4-bytes) for x coordinate, then y coordinate, then z coordinate (12-bytes).
	1-byte value, always 1.
That's it for the vertex chunk.
Next up is the edge chunk.
  4-byte integer for the edge chunk id, always 2.  02 00 00 00
  4-byte integer for the length of the chunk.  In this case: CF 01 00 00
  4-byte integer for the number of edges.  In this case: 33 00 00 00
  for each edge:
    4-byte integer for the first vertex id (id's are assigned in order, starting at 0 as the vertex chunk is read)
	4-byte integer for the second vertex id
	1-byte boolean value for if the edge is smooth or not (00 for sharp, 01 for smooth)
That's it for the edge chunk.
Next up is the face chunk.
  4-byte integer for the face chunk id, always 3.  03 00 00 00
  4-byte integer for the length of the chunk.  In this case: 34 05 00 00
  4-byte integer for the number of faces.  In this case: 13 00 00 00
  for each face:
    float value (4-bytes) for the texture angle
	float value (4-bytes) for the x offset of the texture.
	float value (4-bytes) for the y offset of the texture.
	float value (4-bytes) for the x scale of the texture.
	float value (4-bytes) for the y scale of the texture.
	4-byte integer for the mapping group it belongs to (mapping groups start at 1, not 0, and this value is FF FF FF FF if it has no special material mapping, which is the norm.)
	4-byte integer for the material ID (remember, materials start at 0)
	4-byte integer for the number of INNER edge loops.  (This means loops INSIDE the face, not the border edge loop, that comes later)
	Now comes the border edge loop.
	  4-byte integer for the number of edges that make up the border edge loop.
	  for each edge:
	    4-byte boolean (yes, I said 4, and I said BOOLEAN) for if the edge is flipped. 01 00 00 00 if true, 00 00 00 00 if false.  What is a flipped edge?  Well it's saying instead of edge AB, it would be edge BA.  This is important because edge loops consist of edges, and edges consist of vertices IN A CERTAIN ORDER.  If we have a face with vertices ABCD going clockwise, but have an edge CB instead of BC, edge CB needs to be flipped.
		4-byte integer for the edge id.  As with vertices, edge IDs start at 0 as they are read in.
	If we don't have any inner edge-loops, we move on too the next chunk, otherwise, we list the inner edge loops here.
	For each inner edge-loop
	  4-byte integer for the number of edges that make up this inner loop.
	  for each edge:
	    4-byte boolean for if the edge is flipped.
		4-byte integer for the edge id.
That's it for the face chunk.
Next up is the face-layers chunk.
  This chunk is completely useless.  Even IF you had your geometry/objects inside layers, they DO NOT copy.  But I'll document this just in case.
  4-byte integer for the face-layers chunk id.  Always 06 00 00 00.
  4-byte integer for the length of the chunk.  In this case: 54 00 00 00
  4-byte integer for the number of face-layers... which as far as I can tell is ALWAYS just the same amount as the number of faces.  In this case 13 00 00 00
  4-byte integer for the "format", always 02 00 00 00
  For each face-layer
    4-byte integer, always 00 00 00 00 because the face contains no layer data.  In the .level files, this is a very different story.  Refer to that documentation if you want to learn more.
That's it for the face-layers chunk
Next up is the mapping chunk
  4-byte integer for the chunk ID. 07 00 00 00.
  4-byte integer for the length of the chunk.  In this case: 28 00 00 00.
  4-byte integer for the number of mapping groups.  In this case: 01 00 00 00.
  for each mapping group:
    4-byte integer for the mapping group id (the id's are not auto-generated as they are read-in).
	float (4-bytes) for texture angle
	float (4-bytes) for texture's x scale
	float (4-bytes) for texture's y scale
	float (4-bytes) for texture's x offset
	float (4-bytes) for texture's y offset
	float (4-bytes) for x component of the normal vector
	float (4-bytes) for y component of the normal vector
	float (4-bytes) for z component of the normal vector
That's it for the mapping group
Next up is the geometry group chunk
  This chunk is also completely useless, as groups are not stored in the clipboard.
  4-byte integer for chunk id.  08 00 00 00
  4-byte integer for length of chunk. 0C 00 00 00
  4-byte integer for number of vertex groups. 00 00 00 00
  4-byte integer for number of edge groups. 00 00 00 00
  4-byte integer for number of face groups. 00 00 00 00
That's it for the geometry group chunk, AND that's it for the mesh chunk.

NOW, the object chunk.
It starts the exact same way as the mesh chunk.
  4-byte integer, always 1. 01 00 00 00
  4-byte integer for length of the chunk.  in this case: AD 82 01 00.
  2-byte integer for the chunk type.  01 00
  There is no object count for the object chunk.  We simply read the objects in until there are no more bytes to read.
    Each object:
	4-byte integer, always 1.
	4-byte integer, length of object, in this case E0 80 01 00
	4-byte integer.  Not EXACTLY sure what this is for... but I suspect it's leftover/useless code for layers and/or groups.
	4-byte integer.  Same as above.
	next up is the name of the object.
	4-byte integer for number of characters in the name, and each character is 1-byte each.
	4-byte integer for the number of properties the object has (properties are things like "origin", "color", etc.)
	for each property:
      4-byte integer, always 02 00 00 00, not entirely sure what it is.  A "format" I believe...
	  4-byte integer, length of the property, in this case 22 00 00 00
	  4-byte integer, length of name-string.
		1 byte per character
	  4-byte integer, property_type.  See below for the different property types.  In this case 09 00 00 00 (distance)
	  4-byte integer, number of components (ie a 3 dimensional value, like the origin, has 3 components: xyz).
	  4-byte integer, mystery number, always 0.
	  for each component
		(see "property types" below for what is written for each property)
	If the next value is 00 00 00 00, then the object is finished, BUT if it is 01 00 00 00, then it contains extra data.  The only object that I know of that does this is reflection_probes.
	  4-bytes - length of name string - in this case 18 00 00 00
		1-byte per character for name string - in this case, it is "reflection_proper_cubemap"
	  4-bytes, length of extra data - in this case 00 80 01 00
	  The remaining data is a mystery to me.  The format appears at first glance to be 32 bits per pixel, but I'm not sure.

Property types:
0	String - read in just like with the other strings, but with TWO bytes per character, not 1, and the string length that precedes it is in CHARACTERS, not BYTES.
1	Boolean - a 4-bit integer used as a boolean.  1 = true, 0 = false.
2	Real - float value
3	Integer - integer value
4	File Name - same as String.
5	Color - each color channel is a FLOAT, not a byte.
6	Percentage - float
7	Angle - float
8	Time (pretty sure this is unused) - float
9	Distance - float, in meters (converted to display units by editor)
A	Choice (enumeration) - integer value.

That's it for me today.