Sorting layer and order support for procedural meshes

Link_wtf
Member
Registered: 2014-03-14
Posts: 16

Topic

Unity 4.3 add sorting layers to order sprites easily. It is not specifically designed to be used directly with mesh renderers but it works fine with them has long has the associated shaders have "Zwrite Off" set.


To illustrate my problem, I use several "layers" to draw some sprites (with 2d toolkit) mixed with some tile systems :


From back to front, I draw :
- Background layer (sprites)       <= sorting layer "BackgroundDecor"
- Background tiles (tilesytem)    <= sorting layer "Default", currently not modifiable
- Gameplay layer (sprites)          <= sorting layer "GameplayDecor"
- Gameplay tiles (tilesytem)       <= sorting layer "Default", currently not modifiable


The problem is obvious, I cannot draw anything between the procedural meshes without changing their sorting layers.


The only way (that I know of) to change the layers is to iterate through each chunck and change theirs sorting layers. It's not that bad with a static TileSystem, but more problematic when using runtime painting where I have to check for new Chunks each time I paint a new tile.

I have not implemented this solution yet, I wanted to know if there was a builtin solution and if not if you would consider supporting it.

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 1

I have just created 2 separate tile systems and 'sandwiched' a regular Unity 4.3 sprite between the two and it seems to render in the correct order for me. I have used the Z-axis to introduce a separation between the 2 layers and this appears to work without touching the sorting layers.

In my test project I have the following scene hierarchy:

- Background System [x=0, y=0, z=1]
- Sprite [x=3, y=0, z=0]
- Foreground System [x=0, y=0, z=-1]

That said, I would like to know a little more about the chunk-based workaround which you are suggesting. If I understand correctly you are suggesting that there should be a per tile system property specifying a sort layer which should be used for chunk meshes?

Do you suggest to use the non-documented "sortingOrder" property of the "MeshRenderer" component?

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 2

Okay, I can see the issue which occurs when you actually do need to use sorting layers. The position of the tile system on the Z-axis seems to lose its control over the rendering order.

Link_wtf
Member
Registered: 2014-03-14
Posts: 16

Response 3

I was using the Z position to draw my sprites in the correct order but I now need to reorder them dynamically at runtime and using the position is not very convenient. I can't have any sprite at the same Z pos because our shaders doesn't play nice when 2 or more sprites are drawn at the exact same Z.

By using sorting Layers and the "order in layer" I can easily keep a sorted list and reorder this list on the fly when I reorder a sprite, plus it fixes our shaders problem by setting "ZWrite off" in them.

I have tested updating the sorting layer like this :

if (system != null)
{
	foreach (var chunk in system.Chunks)
	{
		if (chunk != null)
		{
			var mesh = chunk.ProceduralMesh;

			if (mesh != null)
			{
				mesh.renderer.sortingLayerName = SortingLayer;
			}
		}
	}
}

I currently call this each time a new tile is created because I don't know when a new chunk is created but it works even if not very pretty...

I just don't understand why TileSystem.Chunks has a length of 100 while my tile system has only one chunk.

I didn't know that the sorting layers was not documented in unity's doc, it's odd because sortingLayerName and sortingOrder are public properties of UnityEngine.Renderer and work fine with the correct shader... Even 2d Toolkit is using this without any problem.

Anyway, for my specific use, a per TileSystem option that would automatically set new chunk renderers layer would be perfect but I understand that is not a common usage so I could do without.

However would not be against an event raised when a new chunk is created :)

Tanks for your help.

Last edited by Link_wtf (2014-03-14 16:01:50)

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 4

Link_wtf wrote:

I just don't understand why TileSystem.Chunks has a length of 100 while my tile system has only one chunk.

The current layout of the chunks array is itself a map of chunks where [0] represents the first upper-left chunk. By default chunks do not exist for empty areas of a tile system and are thus null. You can of course locate any chunk using the exposed API but the math behind this is chunkRow * chunkColumnCount + chunkRow.

Link_wtf wrote:

I didn't know that the sorting layers was not documented in unity's doc, it's odd because sortingLayerName and sortingOrder are public properties of UnityEngine.Renderer and work fine with the correct shader... Even 2d Toolkit is using this without any problem.

I am currently looking into this and have come across a thread on the 2D Toolkit forum:
http://2dtoolkit.com/forum/index.php?topic=3114.0

Link_wtf wrote:

However would not be against an event raised when a new chunk is created :)

I have added this to my feature request list for you. This seems fairly straightforward and will be included in the next release :)

Link_wtf
Member
Registered: 2014-03-14
Posts: 16

Response 5

Lea Hayes wrote:

You can of course locate any chunk using the exposed API ...

I've missed this one... and it's perfectly adequate as I know the tile index where i painted...


Lea Hayes wrote:

I have added this to my feature request list for you. This seems fairly straightforward and will be included in the next release :)

Even I can do with GetChunkFromTileIndex in this case, that would will not hurt to have it :p
I'll try make you a list of events I need (if its not too much to,ask), there is some stuff I delayed because I did have enough control or information at runtime that could be resolved with some events.

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 6

Link_wtf wrote:

there is some stuff I delayed because I did have enough control or information at runtime that could be resolved with some events.

Excellent! perhaps you could post these in a separate topic?

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 7

I have just learnt that you can change the transparency sorting mode of the camera itself. When this is set to "Orthographic" it seems to play nice with both tile systems and sprites where you use the Z-axis to control depth.

You might want to take a look at the following post:
http://forum.rotorz.com/viewtopic.php?pid=65#p65

I would be very interested to know whether this helps with your scenario as well.

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 8

I am too having issue with sorting layer options missing from Rotorz. I use it extensively for my sprite drawing order and it is really pain to use my puny editor script to fix it again and again.

It would be super cool to be able to select sorting layer name and order using the brush menu, just like how we can set up layer and tags.

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 9

BTW, just found that layer setting for procedural generated tiles doesn't work. It always set it to default no matter what the brush setting is. Tag setting is also broken. Always comes out as untagged.

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 10

pixellore wrote:

I am too having issue with sorting layer options missing from Rotorz. I use it extensively for my sprite drawing order and it is really pain to use my puny editor script to fix it again and again.

The ability to customize sorting layers for renderer components became a documented feature of Unity 4.5 which means that it can now be added to Rotorz Tile System. I cannot provide a release date, but it will be soon.

pixellore wrote:

It would be super cool to be able to select sorting layer name and order using the brush menu, just like how we can set up layer and tags.

Sorting will be specified on a per tile system basis rather than on a per brush basis.

pixellore wrote:

BTW, just found that layer setting for procedural generated tiles doesn't work. It always set it to default no matter what the brush setting is. Tag setting is also broken. Always comes out as untagged.

Layer and tag are not applied to the procedurally generated mesh. In the future the procedurally generated game object may be entirely hidden from the hierarchy window (like with nGUI and Unity 4.3 sprites).

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 11

I can understand that for procedurally generated tile map, its sorting layer options may need to be per tile map basis. But for other "normal" brushes that uses prefab either has to be per brush basis or must have option to not alter (force) them.

Also for procedurally generated tile map, tag and layer setting is also important, because it is possible to use those information in the game. For example you may want to tag it or make the layer as "ground" or "roof" and then you can have some custom rendering system that only special camera needs to render etc. Is there any reason why tags and layers are not set at all? Unity camera rendering system uses layers extensively and tags are good for detecting types of objects (tile maps) , it is pretty useful feature.

I can't see what can possibly go wrong for setting tags and layers to the tile brush's setting? (or any other disadvantages for that matter)

Last edited by pixellore (2014-06-20 15:53:33)

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 12

pixellore wrote:

I can understand that for procedurally generated tile map, its sorting layer options may need to be per tile map basis. But for other "normal" brushes that uses prefab either has to be per brush basis or must have option to not alter (force) them.

Sorting options will initially only be applied to procedurally generated tile meshes. If you want to apply sorting to prefabs or non-procedural tiles then you will need to attach a custom script to achieve this.

pixellore wrote:

I can't see what can possibly go wrong for setting tags and layers to the tile brush's setting? (or any other disadvantages for that matter)

Procedural tiles are represented using a single procedurally generated mesh (one per chunk). This is beneficial since one game object can represent many tiles (from many brushes).

Since the procedurally generated mesh is just one game object (i.e. to improve batching, reduce overhead of many separate game objects) there is only one "tag" and one "layer" property. However, since the procedurally generated mesh doesn't contain any special components, these values should not influence gameplay.

Non-procedural tiles (and Unity sprites) require a minimum of one game object per tile. Whilst this provides a greater degree of flexibility, there is additional overhead. Procedural tiles are quite static in nature whereas sprites are dynamic.

You can attach prefabs to procedural tiles which can have a custom tag and layer if desired. This is relevant when attaching colliders, game objects, etc.

Since the only way to address a procedural tile is via GetTile you may want to consider using flags to identify custom state at runtime.

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 13

Like i have mentioned before. Tags and layers per tile map basis is still useful since it can be used for many other purposes such as custom rendering pipeline. By tags and layers on procedurally generated tiles i was refering to the whole tile map not individual tiles. By being able to specifiy say a layer for specific tilemap i can use it on my special camera render layer to only render that tilemap do special things on them, such as render it to the render texture and do overlay post processing etc..

So ability to specifiy tags and layer for whole tile map still presents good use for it.

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 14

pixellore wrote:

By being able to specifiy say a layer for specific tilemap i can use it on my special camera render layer to only render that tilemap do special things on them, such as render it to the render texture and do overlay post processing etc.

It doesn't make sense to apply brush tag/layer to the procedural mesh. Which brush should the procedural mesh listen to when multiple brushes define different tag/layer.

Instead this should be defined on a per tile system basis. You can adjust the tag and layer for the tile system already using the inspector which should allow you to do just that. If this doesn't work automatically, then perhaps it would make sense for the procedural mesh to inherit the tile system tag and layer.

Last edited by Lea Hayes (2014-06-21 02:04:16)

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 15

Well.. I know I can use Unity's inspector and select the root + change childern layer setting etc.. but when you "paint" tiles, it generates the chunks on the fly. so the new ones are not set automatically. Whenever I make changes, add tiles to the tile map, then I have to do it again and again just to make sure the new ones are set to the correct layers , tags.

This is what I was going on about. I have to repeatively do this everytime I do new painting. I was hoping Rotorz will some how manage this by having tilemap tag + layer setting and then apply it to newly generated meshes automatically for me.

Perhaps make the root of the tilemap object the setting of the meshes it generate?

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 16

Lea Hayes wrote:
pixellore wrote:

By being able to specifiy say a layer for specific tilemap i can use it on my special camera render layer to only render that tilemap do special things on them, such as render it to the render texture and do overlay post processing etc.

It doesn't make sense to apply brush tag/layer to the procedural mesh. Which brush should the procedural mesh listen to when multiple brushes define different tag/layer.

Instead this should be defined on a per tile system basis. You can adjust the tag and layer for the tile system already using the inspector which should allow you to do just that. If this doesn't work automatically, then perhaps it would make sense for the procedural mesh to inherit the tile system tag and layer.

I agree. That should do the trick.

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 17

pixellore wrote:

This is what I was going on about. I have to repeatively do this everytime I do new painting.

You can hook into the PaintingUtility.TilePainted event to automate custom functionality both in-editor and at runtime.

pixellore
Member
Registered: 2014-05-29
Posts: 69

Response 18

I see , so I should attach a.. custom script to the root of the tile map and hook it into the Brush.TilePainted event and set the mesh's tag and layer value to as I see fit?

Lea Hayes
Rotorz Limited
From: United Kingdom
Registered: 2014-03-04
Posts: 638

Response 19

pixellore wrote:

I see , so I should attach a.. custom script to the root of the tile map and hook it into the Brush.TilePainted event and set the mesh's tag and layer value to as I see fit?

You can perform any custom logic...

Though I like the idea of the procedural mesh inheriting the tile system layer... This is a simple change so I will do you a build tomorrow sometime to experiment with :)