Blender -> Spark .level exporter! :D
BeigeAlert
Texas Join Date: 2013-08-08 Member: 186657Members, Super Administrators, Forum Admins, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, NS2 Map Tester, Reinforced - Diamond, Reinforced - Shadow, Subnautica Playtester, Pistachionauts
Texas Join Date: 2013-08-08 Member: 186657Members, Super Administrators, Forum Admins, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, NS2 Map Tester, Reinforced - Diamond, Reinforced - Shadow, Subnautica Playtester, Pistachionauts
MAJOR UPDATE!
The old version of the plugin is preserved below, but for now, please use this new version, found in this thread: http://forums.unknownworlds.com/discussion/136044/blender-spark-addon-for-blender
The old version of the plugin is preserved below, but for now, please use this new version, found in this thread: http://forums.unknownworlds.com/discussion/136044/blender-spark-addon-for-blender
Hey everybody!
Using information from this thread: forums.unknownworlds.com/discussion/130973/map-format-documentation
I was able to create a basic exporter addon for Blender that will spit out a .level file!
I've attached a .zip with the addon scripts for blender, but here's an alternative link: https://www.dropbox.com/s/50b5r2pblhqqcr5/io_scene_spark_level.zip
To install: navigate to your addons folder inside the program files directory (mine is "C:\Program Files\Blender 2.69\2.69\scripts\addons") and it should just extract right to in there. You should see plenty of other "io_scene_blah blah" plugins already there that should clue you in to how it should look after installing.
There is no "importer" provided, but you should be able to just use the .obj option from the spark editor to get it into Blender.
Now, I suppose a feature list is in order:
-Exports all meshes, or the selected object meshes to a spark *.level file.
-If you want an edge to be set to smooth in spark, you can use "mark seam" in blender to make it so. (yea yea, I know there's also a "set smooth" option in blender, but I couldn't get that working)
To do:
-Figure out some way to get materials to work. As it is right now, all the faces default to "materials/dev/dev_1024x1024.material", which is just the default greybox texture. All texture coordinates are reset to 1.0 scale in each dimension, 0.0 offset in each dimension, and 0 rotation. I'm not sure what the best way would be to include the textures in the exporter. I'll need to get better acquainted with Blender python... I just barely scraped by with this one!
-I've been considering making a sort of "auto-rotation" for the materials on the faces, so you can easily apply trim, but this might be more trouble than it's worth. This would, of course, come after any material system is thought up/ implemented.
NOTICE: Spark faces will APPEAR GLITCHED OUT if they are too small (ie any edge is <1.0" in length). This is nothing to do with the exporter. All you need to do to fix this is use bigger faces. If you absolutely MUST have smaller faces... use props.
Also, just a disclaimer: I have tried my best to ensure that this software is safe, but I'm only human (and a crappy programmer at that), so use at your own risk.
That being said, please report any defects, and feel free to suggest features!
Technical talk below, spoiler tags provided for your convenience.
Using information from this thread: forums.unknownworlds.com/discussion/130973/map-format-documentation
I was able to create a basic exporter addon for Blender that will spit out a .level file!
I've attached a .zip with the addon scripts for blender, but here's an alternative link: https://www.dropbox.com/s/50b5r2pblhqqcr5/io_scene_spark_level.zip
To install: navigate to your addons folder inside the program files directory (mine is "C:\Program Files\Blender 2.69\2.69\scripts\addons") and it should just extract right to in there. You should see plenty of other "io_scene_blah blah" plugins already there that should clue you in to how it should look after installing.
There is no "importer" provided, but you should be able to just use the .obj option from the spark editor to get it into Blender.
Now, I suppose a feature list is in order:
-Exports all meshes, or the selected object meshes to a spark *.level file.
-If you want an edge to be set to smooth in spark, you can use "mark seam" in blender to make it so. (yea yea, I know there's also a "set smooth" option in blender, but I couldn't get that working)
To do:
-Figure out some way to get materials to work. As it is right now, all the faces default to "materials/dev/dev_1024x1024.material", which is just the default greybox texture. All texture coordinates are reset to 1.0 scale in each dimension, 0.0 offset in each dimension, and 0 rotation. I'm not sure what the best way would be to include the textures in the exporter. I'll need to get better acquainted with Blender python... I just barely scraped by with this one!
-I've been considering making a sort of "auto-rotation" for the materials on the faces, so you can easily apply trim, but this might be more trouble than it's worth. This would, of course, come after any material system is thought up/ implemented.
NOTICE: Spark faces will APPEAR GLITCHED OUT if they are too small (ie any edge is <1.0" in length). This is nothing to do with the exporter. All you need to do to fix this is use bigger faces. If you absolutely MUST have smaller faces... use props.
Also, just a disclaimer: I have tried my best to ensure that this software is safe, but I'm only human (and a crappy programmer at that), so use at your own risk.
That being said, please report any defects, and feel free to suggest features!
Technical talk below, spoiler tags provided for your convenience.
What the exporter does is write out the vertex, edge, face, and material chunks, and put those inside the mesh chunk. Other features of Spark are left out, such as entities, mapping groups, geometry groups, triangles, face layers, etc. What, triangles? Yea I don't know what that is for either, but it seems to have little to do with the actual geometry. My best guess is that it is how the engine has triangulated the faces in the editor.
When I started writing this script a few days ago, I went through the forum post with the documentation in it. I had to fill in a few holes here and there with educated guesses, but eventually I got the hang of it. (Now I can read the mesh chunk in hexadecimal without needing a cheat sheet!)
I started out by dissecting a simple .level file... a cube I think it was. After studying the file alongside the format guide, I decided to start removing chunks, seeing what the editor could do without. I found that all the editor needs to load the geometry data is the mesh chunk, and inside this, all it needs is the vertex, edge, and face chunks.
FINALLY, after much screaming and cursing at my computer screen for throwing up error messages when the single lines of code went through python IDLE just fine, but for whatever reason Blender had an issue with (damn Blender...
), I FINALLY had something that would export. I open it up in the editor aaaannndd... oh huh... that's weird looking. Some of the faces are there, but most aren't. Why is that? First, a little bit of background on how the geometry is written.
First off, you have your vertex chunk. This contains the positions of each vertex in the mesh. Each one is assigned an index based on the order it is created.
Next up, you have the edge chunk. This contains a list of all the edges, with each edge itself being a list of two vertices. The edge chunk doesn't have any coordinates, only 32-bit unsigned integers corresponding to the indices of the vertices that the edge connects. Like vertices, edges are assigned an index based on the order it is written into the file. ie edge 0 might connect vertices 3 and 7. This is NOT the same as edge 0 connecting vertices 7 and 3. The order is very important, and this is what tripped me up for a while.
Finally is the face chunk. This contains, in addition to material settings and layer/group settings, references to all the edges that make up a face. Luckily I didn't have to worry about multiple edge loops in a single face (ie a hole in a wall) as Blender doesn't have this capability. So for my purposes here, a face is made up of one edge loop, and an edge loop is made up of n edges. The way the Blender methods work, the order the edges are spit out is correct, but NOT the orientation! Say you had a square face with corner vertices (clockwise) labeled ABCD. You would have 4 edges for this face, AB, BC, CD, DA. BUT, you could ALSO have edges BA, CB, DC, AD, and the order of the edges wouldn't change, but each one would need to be flipped. As it turns out, the spark level format has a field for just this purpose. To define the edge loop for the face, you list off each edges' id. But, and this puzzled me at first, before each edge, you also have another 4 (???) bytes to indicate if the edge is flipped or not. (four, really???) Anyways, I just made them always 0 at first, thinking it was something unused left in the code, and quickly realized this was a mistake. I exported that... thing... you see above in the screenshots. The edges and vertices made it just fine, but the faces were weird. Eventually I realized the importance of this field, and was able to get it working. If you made it this far reading this, congratulations! --Beige
When I started writing this script a few days ago, I went through the forum post with the documentation in it. I had to fill in a few holes here and there with educated guesses, but eventually I got the hang of it. (Now I can read the mesh chunk in hexadecimal without needing a cheat sheet!)
I started out by dissecting a simple .level file... a cube I think it was. After studying the file alongside the format guide, I decided to start removing chunks, seeing what the editor could do without. I found that all the editor needs to load the geometry data is the mesh chunk, and inside this, all it needs is the vertex, edge, and face chunks.
FINALLY, after much screaming and cursing at my computer screen for throwing up error messages when the single lines of code went through python IDLE just fine, but for whatever reason Blender had an issue with (damn Blender...
), I FINALLY had something that would export. I open it up in the editor aaaannndd... oh huh... that's weird looking. Some of the faces are there, but most aren't. Why is that? First, a little bit of background on how the geometry is written.First off, you have your vertex chunk. This contains the positions of each vertex in the mesh. Each one is assigned an index based on the order it is created.
Next up, you have the edge chunk. This contains a list of all the edges, with each edge itself being a list of two vertices. The edge chunk doesn't have any coordinates, only 32-bit unsigned integers corresponding to the indices of the vertices that the edge connects. Like vertices, edges are assigned an index based on the order it is written into the file. ie edge 0 might connect vertices 3 and 7. This is NOT the same as edge 0 connecting vertices 7 and 3. The order is very important, and this is what tripped me up for a while.
Finally is the face chunk. This contains, in addition to material settings and layer/group settings, references to all the edges that make up a face. Luckily I didn't have to worry about multiple edge loops in a single face (ie a hole in a wall) as Blender doesn't have this capability. So for my purposes here, a face is made up of one edge loop, and an edge loop is made up of n edges. The way the Blender methods work, the order the edges are spit out is correct, but NOT the orientation! Say you had a square face with corner vertices (clockwise) labeled ABCD. You would have 4 edges for this face, AB, BC, CD, DA. BUT, you could ALSO have edges BA, CB, DC, AD, and the order of the edges wouldn't change, but each one would need to be flipped. As it turns out, the spark level format has a field for just this purpose. To define the edge loop for the face, you list off each edges' id. But, and this puzzled me at first, before each edge, you also have another 4 (???) bytes to indicate if the edge is flipped or not. (four, really???) Anyways, I just made them always 0 at first, thinking it was something unused left in the code, and quickly realized this was a mistake. I exported that... thing... you see above in the screenshots. The edges and vertices made it just fine, but the faces were weird. Eventually I realized the importance of this field, and was able to get it working. If you made it this far reading this, congratulations! --Beige
Comments
Blender can do many things that spark can't. Insetting an edge on the edge of a face for trim, for example. You probably wouldn't want to send your whole level over at once to blender, but you could send small pieces over to blender individually, then copy+paste them from one editor window back into your main level.
One feature I keep thinking back to is booleans. Blender has booleans, but spark doesn't. So you could use Blender to make some pretty interesting geometry. Not everything should be turned into a prop at the end... though even if you do, you can use Blender to help create some geometry around that prop to better integrate it into the scene.
Also, I've found I can make a level much faster in blender than I can in Spark, so you could potentially use this for rapid prototyping.
Completely agree.
Imagine the Q3 sound mod for each bullet.
*A sphere (Just that should blow any resistance) Domination
*A terrain like Biodome (Who said Kodiak ?). Displacement tool is just a pain. Rampage
*Every round volume like a Fancy Star trek space ship. god like!
*Every new props Lu lu lu ludicrous stuff
*Plus you don't have to rely on editor to correct something, you just re-export. And then limitations get away.Unstoppable!!!
Get it ?
@BeigeAlert :
Nice shot son! Never let someone discourage you. If no dev helps on this which would be a real SHAME (because he has a POC), just ask me. I have, let's say some knowledge on coding and data structure ( Who said Knuth? ), maybe i can help.
The only problem is every time you save to .obj, everything gets triangulated, which is typically a quick fix in blender (ctrl + a, alt+j to fill faces), but sometimes gets screwey. However, just last night I used the booleans to carve away a tunnel through the debris pile in temple. It worked pretty well, needs a bit of cleanup, but I can walk around inside it, and it's friggen sweet! (new build published soon, btw.) With geometry that random and dense, there is simply NO WAY to do a boolean operation like that in spark. I made the broken pillars and broken walkway (royal chambers) by just drawing lines over the geo and filling in the holes, but it would have looked much better if I had had booleans to work with.
And yea, like you mentioned, the displacement mapping is really awful inside of spark... BUT... I'd like to share some bizarre behavior I noticed. If you use displacement on a plane, then try to edit the texture... you can't! The values change, and you can swap out the texture, but it's as if the texture on that polygon is referencing a set of UV coordinates, instead of spark's built in texture system's values. Seriously, make a plane, use the displacement tool on it a little bit, build the geometry, then scale it waaay up on the y axis. The texture stays connected. I will try to investigate what this looks like in the .level format, see if I can get uv coordinates from Blender into Spark.
My guess is this behavior post-dates the level documentation post, so we won't see any trace of it. I figure it's a completely undocumented chunk being created somewhere... gotta figure out how the face references this new data though... hrm...
I didn't analyze the blender format but there will be always be the same stuff concerning 3D. Vertex, faces, polygons (planar set of triangles), uv maps, etc. Same for editor but a data is a data, no matter what.
See it as direction to decipher it.
Blender
If i was a blender developer that is working in a "object environment" i would:
-Organize things so i get a hierarchy that is comfortable like this : scene > element (object, camera etc..) > Position, rotation, scale variables and sub element variables depending on type. Nesting is an easy logic and works well as long as you clean up things properly when needed.
UV map way of doing things in blender is the latest method available (unwrap). Not the 3DS3 style (UV maps with gizmos like cylinder, sphere etc..). The old cylinder UV map is available but no gizmos available. It's a bit cranky (aligned on view) but not a problem for your research.
So there are different UV map available for any objects in a scene. Then comes the material and "nodes". It's the key word you gotta catch in order to imagine what's behind this engine (Cycles). Ho and it will probably be different depending on the engine you use in blender. Try to stick with Cycles as it is the one that will prevails (apparently).
The editor
The editor has a way of doing geometry that can be called classic. Everything is a triangle in the end (show triangulation). UV Mapping is planar (rectangle). Though the way of using it isn't common (multiplier factor for U&V) it's ok. No cylinder or sphere UV map. But you know that.
So let's get to business...
Analysis
In blender you probably noticed that you had a UV map editor that allows you to change the way a picture is applied on a face/polygon. Basically this is a 2D projection from the 3D faces on the image you see in the editor. And you noticed that the 2D polygons can be changed and look totally different from the face it belongs to. It means it isn't interdependent. So it's linked with a code to link them together.
So, for a cube you have 6 UV polygons linked to each faces of the cube. Again hierarchy is helpful for storage and linking.
Many forms can be forged:
Several arrays, one for 3D object, one for UV polygons, one for linking. More like a database form.
Or:
1 array storing the object, then the faces, and inside the faces object (can be class), the necessary data for UV mapping.
That suggest you have to scan your blender scene to catalog nodes. If i were you i would scan as much as possible with your script. A simple "dump" in XML format would do. Else Text format is enough.
You'll have to find UV map information, material information (which aren't properly exported in any export format using Cycles), 3D information. Once you have this you have to convert/adapt it for editor. Watch out for image path which would be static when exporting (not relative).
Investigation
Make 3 scenes in blender. Remove camera and light.
A/ 1 cube only, no UV map, no material
B/ 1 cube only, unwrap UV map, 1 image material
C/ 1 cube only, planar UV map, 1 image material
Then you can start to analyze your scene using your script and dump some. Comparing the dump files will surely give you the answer you seek.
If you don't have the path to the image, you missed something. A ChildNode for example.
I can't write a book here. Hope it helps.
I think you misunderstood me.
I am very well versed in pretty much every inch of Blender, (except scripting! :P)
When I was talking about UVs, I was talking about in Spark for surfaces that had used the displacement tool. The way textures work in spark has little to no resemblance to UV mapping... UNTIL you use the displacement too, then all of a sudden, the texture coordinates distort as if their UVs are locked.
Oooh yea, that's probably not going to happen.
Also, I'm going to try to figure out how to get a workflow to where you can export to blender using the clipboard, rather than having to save a million billion files.
Are you exporting from the menu, or are you just trying to run the script? It should work as an addon.
I have to load and run "__init__.py" in the script console in order to see it in the menu.
Win64 / Blender 2.70a
in:
C:\Program Files\Blender Foundation\Blender\2.70\scripts\addons\io_scene_spark_level\
From the export menu or running export_spark_level.py in console does the same.
SystemError: Parent module '' not loaded, cannot perform relative import
Some custom script dependency ?
Should be able to just activate it from the addon menu. Is it not showing up there? File -> User Preferences, category filter "Import-Export", and it's called "Natural Selection 2 Spark Editor Geometry Export"
Also, what version of Blender are you using?
My bad. Working now.
Still:
You have a issue with triangulation.
The editor isn't a genius with triangulation. So it gives this :
and
(corner of a beveled cube)
Applying or not the modifier on the mesh is the same.
EDIT: I just tried it out, and it seems to work just fine. The only thing I can think of is the scale issue.
EDIT2: AHAHA!!! Yup that's what it is. You've got your faces too small. I was just now able to repro your pictures exactly by scaling the object up and down.
If i change the dimensions of a cube, it automatically change the scale factor to 50 from a cube of length 2 (scale factor 1). Also exporting it makes a 64 (in length is seems) cube in the editor.
Blender:
Editor:
Any step by step procedure ?
Yea I screwed up here. What's happening is it's using the local coordinates rather than the global coordinates, so it doesn't matter how much you scale up the object, that information is ignored... for now...
Blender units are meters, so you'll have to either switch it to imperial or scale it at the end. I recommend scaling it, as using imperial in blender is a baffling ordeal. Everything goes off-grid no matter how precise you are with your conversion factors... and typing in values manually is a pain when you have to supply the unit at the end every time.
I screwed up when writing the script, so all the transformations are in LOCAL space, not WORLD space. So that means before you export, the object's origin needs to be at (0,0,0), the scale needs to be (1.0,1.0,1.0) and the rotation needs to be (90,0,0) since Blender and Spark have different "up" axes.
To do this: Apply the scale (Ctrl+A -> scale), set the origin (Shift+C to set cursor to 0,0,0, then Tools -> Transform -> Set Origin -> Origin to 3D Cursor), then set the rotation (rotate the OBJECT 90 degrees on x axis, around 3d cursor, THEN rotate the MESH of that object -90 degrees on the x axis, around the 3d cursor).
I know this is a huge pain in the butt. I'll try to find some time later to improve the script so transformations are applied before the export... and maybe add in an option to convert meters/blender-units to inches.
See below.
There's two new check-boxes for you: "Unit Correction" and "Literal Axes". The former means it'll automatically scale your scene down to 1/39.37 (inches to meters conversion). Don't use this if you've ALREADY been working in inches with blender... this is designed to allow you to force the exporter to see blender units and spark inches as being 1:1... when normally the conversion is 1:39.37. So basically, if the checkbox is on, it'll assume 1 blender unit, is 1 spark inch. If it's off, it assumes 1 blender unit is 1 spark METER.
Literal Axes means to write out the data where XYZ is just plain old XYZ... no special conversions. You probably don't want this checked. When it's un-checked, it automatically does the conversions so that you don't need to pre-rotate your scene. (Blender -> Spark = +X -> +X, +Z -> +Y, +Y -> -Z) Honestly, I can't think of a single reason why you would even use this checkbox... oh well. Regardless, you don't have to worry about rotating your objects around anymore.
EDIT: Forgot to point out: all you have to do is re-download from the link in the OP. The link stays the same.
How do the units convert? for example from blender to unreal. 1 blender unit is 1 metre in blender, but 1 blender unit is 1 cm in unreal engine 4, so I have to model at ridiculous scale which does indeed cause a couple of rendering bugs(partially transparent edges on curved surfaces in blender). Not really a big issue.
Also Unreal engine 4 recommends manually triangulating a mesh before sending it to the editor. Would you recommend manually triangulating a map before sending it to spark? Or is the intention that you import without triangulation so you can keep working in spark?
Note: When I'm talking about ue4, I'm referring to static mesh workflow. When I'm talking about ns2, I'm talking about blender -> spark .level information. I realise there's a big difference between the two workflows, hence why I am asking these questions.
Oh man I don't have a clue about the units in UE4. There are two options for the units to export with my script. The default ("unit conversion" = true) means that 1 BU becomes 1 inch (since most people seem to work in inches in Spark). The other way ("unit conversion" = false) means that 1 BU = 1 meter, meaning if you are working in spark in inches... you're going to get a lot of seemingly arbitrary measurements. Note that if you've setup Blender to use imperial, you should NOT use "convert units"... though a word to the wise: keep it at BU. It's a HUGE pain in the ass to get Blender to work in imperial AND have the grid imperial without having the accuracy drift (ie move it 1200 units, and it somehow ends up 1200.0001 units away).
But as for your question about UE4 units... not sure. You'll need to just figure out what the conversion factor is, if any. Remember, for ns2, you'll want your doorways of your main routes to be roughly 192-256 units wide (really depends on context, map playstyle... etc. etc.)
As for triangulating the mesh... not necessary at all. The key difference between how Blender and Spark store faces is that while Blender has NGons (>4 verts/edges per face), Blender does NOT have the inner-edge loop functionality of spark (ie you can't make a donut shape in blender with a single face, you'd have to break it up into two faces). In fact, it's this key difference that has been the main obstacle to me creating a Spark level IMPORTER for Blender (how would I deal with holes in the mesh??? I don't know the complicated algorithms involved in generating the triangles so that they don't pass over the hole). I think I've actually got an idea for how I can get this to work now, but it will be a while before I have the free time to actually do this. That, and there's not much point in writing an importer unless I can figure out how to get textures to work as well... so there's that too.
If you have a lot of undesired tessellation in your geometry and you want to clean it up, what you can do in Blender is use the Decimate modifier, set it to "planar" and tweak the angle threshold to the appropriate amount. You can also remove edges but keep the face by using the "dissolve edge" command under the X menu.
Yay! Thanks Ghoul.
As far as my understanding goes, setting blender to inches won't actually make it work in inches, it'll just give you inches to work with but it's still working in blender units.
Oh nice, didn't realise i could set the decimate modifier to planar.
I tried to manually create an ngon tube in blender, it didn't work. Is spark doing a genuine curve(like id tech 3) or is it just using ngons and then triangulating once it gets to game (all game engines triangulate all faces once in game, unless it's a curve in id tech 3).
No such thing as curves in spark, or in any game engine (that I know of). Id Tech 3 just had curved brushes. At the end of the day, all that's doing is just automatically generating your many straight edges that make up the curve automatically.
The only thing currently preventing me from going through with this is I have yet to figure out an elegant way to store texture data PER FACE in Blender.
See, here's how the problem breaks down: In Spark, a face's texture contains 6 properties: Scale X, Scale Y, Shift X, Shift Y, Rotation, as well as the material that is assigned to it -- I'll be referring to this as "Spark tex params" from now on. Blender maps textures to faces using UV coordinates, not the 6 properties listed above. Converting from spark tex params to uv coordinates isn't much of a problem, it's converting BACK that will be the real issue. So, I'm thinking the way to go is to have each face retain its spark tex params, and have it re-calculate the projection any time that face is changed. This presents two problems: 1) how to store per-face data without preventing users from modifying the mesh, and 2) how to get Blender to report any mesh changes so the textures can be updated???
So I'm opening the floor up to suggestions here... because I'm out of ideas.