Marshalling C# structures into Direct3D 11 cbuffers using SharpDX

UPDATE 08/03/2011

• I have incorporated Alexandre Mutel's refactoring suggestion from his comment below, to avoid creating a DataStream every time.
• He also suggests an alternative approach, which is more elegant but requires more code.

It's tempting, after Googling a problem, finding no solutions, then going old-school and actually trying to solve it myself, and eventually coming up with a solution, to keep running and never look back. But this particular problem gave me lots of SEO-friendly keywords to put in the page title of a blog post, so here we are.

Background

A bit of background - I'm working on some enhancements to DotWarp, which will allow an arbitrary number of directional lights, point lights, and spotlights to be applied to a scene. DotWarp uses SharpDX to wrap Direct3D 11. DotWarp works with a single material type, defined in a file called BasicEffect.fx. I used that name because it roughly follows XNA's BasicEffect API (or rather it did until this most recent change to support arbitrary numbers of different light types).

The .fx extension is also misleading, because I'm not using the Direct3D 11 Effects framework - instead I compile the vertex and pixel shaders separately. I have a BasicEffect class defined in C# that tries to abstract that fact away from callers.

The problem

Direct3D 11 shader parameters are declared inside cbuffer structures (parameters declared in global scope will be added to an implicit cbuffer). cbuffers themselves are outside the scope of this blog post, but they're pretty cool and offer some good performance improvements over the Direct3D 9 way of doing things. A typical cbuffer might look like this:

cbuffer BasicEffectVertexConstants : register(b0)
{
    matrix WorldViewProjection;
    matrix World;
}

The difficulty comes in setting the values for WorldViewProjection and World from your C# code. It gets even tricker if you want to use arrays inside your structures.

A solution

What follows is fairly SharpDX-specific, although the same general principle should work for SlimDX too. First, we need to define a C# struct that matches the HLSL struct.

[StructLayout(LayoutKind.Explicit, Size = 128)]
internal struct BasicEffectVertexConstants
{
    [FieldOffset(0)]
    public Matrix WorldViewProjection;

    [FieldOffset(64)]
    public Matrix World;
}

You will notice I've used attributes to explicitly set the size and field offsets. This is to avoid differences between .NET and HLSL in how fields are packed. MSDN has a good page on Packing Rules for Constant Variables, which covers the HLSL side. For the struct above, it would be packed correctly for HLSL even without the attributes, but that isn't always true, so I prefer to be consistent and always set the size and field offsets explicitly.

Now we need a couple of helper methods. The first is going to create a Direct3D 11 constant buffer resource, for a particular struct type. The second helper method is going to update that Direct3D 11 constant buffer resource with new values. UPDATE To keep things simple, we'll create a new class:

internal class ConstantBuffer<T> : IDisposable
    where T : struct
{
    private readonly Device _device;
    private readonly Buffer _buffer;
    private readonly DataStream _dataStream;

    public Buffer Buffer
    {
        get { return _buffer; }
    }

    public ConstantBuffer(Device device)
    {
        _device = device;

        // If no specific marshalling is needed, can use
        // SharpDX.Utilities.SizeOf<T>() for better performance.
        int size = Marshal.SizeOf(typeof (T));

        _buffer = new Buffer(device, new BufferDescription
        {
            Usage = ResourceUsage.Default,
            BindFlags = BindFlags.ConstantBuffer,
            SizeInBytes = size,
            CpuAccessFlags = CpuAccessFlags.None,
            OptionFlags = ResourceOptionFlags.None,
            StructureByteStride = 0
        });

        _dataStream = new DataStream(size, true, true);
    }

    public void UpdateValue(T value)
    {
        // If no specific marshalling is needed, can use 
        // dataStream.Write(value) for better performance.
        Marshal.StructureToPtr(value, _dataStream.DataPointer, false);

        var dataBox = new DataBox(0, 0, _dataStream);
        _device.ImmediateContext.UpdateSubresource(dataBox, _buffer, 0);
    }

    public void Dispose()
    {
        if (_dataStream != null)
            _dataStream.Dispose();
        if (_buffer != null)
            _buffer.Dispose();
    }
}

Armed with this class, we can start to write our actual code. First, we'll create the constant buffer. This should be in your initialisation code:

_vertexConstantBuffer = new ConstantBuffer<BasicEffectVertexConstants>(device);

Don't forget to Dispose() of that instance in your cleanup code.

Finally, we can update the values inside the cbuffer using the values from our C# struct with this code. Obviously you'll want to replace the structure type, and the setting of the field values, with your own parameters.

var vertexConstants = new BasicEffectVertexConstants();

Matrix wvp = ConversionUtility.ToSharpDXMatrix(World * View * Projection);
vertexConstants.WorldViewProjection = Matrix.Transpose(wvp);
vertexConstants.World = Matrix.Transpose(ConversionUtility.ToSharpDXMatrix(World));

_vertexConstantBuffer.UpdateValue(vertexConstants);

DeviceContext.VertexShader.SetConstantBuffer(0, _vertexConstantBuffer.Buffer);

Field offsets

If your cbuffer is more complicated, then you'll have some other problems. Take this example:

cbuffer BasicEffectPixelConstants : register(b0)
{
    float3 CameraPosition = float3(0, 5, 20);

    bool LightingEnabled = true;

    float3 AmbientLightColor = float3(0.3, 0.3, 0.3);

    float3 DiffuseColor = float3(0.1, 0.7, 0.1);
    float3 SpecularColor = float3(1, 1, 1);
    float SpecularPower = 16;

    bool TextureEnabled = false;

    float Alpha = 1;
}

If you tried to map that onto a C# struct without explicitly setting field offsets, it wouldn't work, because HLSL has very strict rules for packing constant variables, which are not the same as .NET. I haven't found an automatic way of calculating the correct field offsets, but I have setup unit tests that allow me to check that I got it right. In this case, the corresponding C# struct looks like this:

[StructLayout(LayoutKind.Explicit, Size = 80)]
internal struct BasicEffectPixelConstants
{
    [FieldOffset(0)]
    public Vector3 CameraPosition;

    [FieldOffset(12)]
    public bool LightingEnabled;

    [FieldOffset(16)]
    public Vector3 AmbientLightColor;

    [FieldOffset(32)]
    public Vector3 DiffuseColor;

    [FieldOffset(48)]
    public Vector3 SpecularColor;

    [FieldOffset(60)]
    public float SpecularPower;

    [FieldOffset(64)]
    public bool TextureEnabled;

    [FieldOffset(68)]
    public float Alpha;
}

Arrays

Finally, I can cover the case that prompted this blog post: using arrays inside your structs. You'll want to do this, for example, if you want to have a collection of lights, and loop through them in your shader code.

I have a DirectionalLight structure in HLSL:

struct DirectionalLight
{
    bool Enabled;
    float3 Direction;
    float3 Color;
};

... for which I've defined the corresponding structure in C#:

[StructLayout(LayoutKind.Explicit, Size = 32)]
internal struct BasicEffectDirectionalLight
{
    [FieldOffset(0)]
    public bool Enabled;

    [FieldOffset(4)]
    public Vector3 Direction;

    [FieldOffset(16)]
    public Vector3 Color;
}

I then have this cbuffer:

#define MAX_LIGHTS 16

cbuffer LightConstants : register(b1)
{
    int ActiveDirectionalLights;
    DirectionalLight DirectionalLights[MAX_LIGHTS];
}

... which maps to this C# structure:

private const int MaxLights = 16;

[StructLayout(LayoutKind.Explicit, Size = 4 + (32 * MaxLights) + 12 /* padding */)]
internal struct BasicEffectLightConstants
{
    [FieldOffset(0)]
    public int ActiveDirectionalLights;

    [FieldOffset(16), MarshalAs(UnmanagedType.ByValArray, SizeConst = MaxLights)]
    public BasicEffectDirectionalLight[] DirectionalLights;
}

The additional MarshalAs attribute on the array is the key to getting this to work. Without that, .NET won't know how to map the structure into the fixed size bit of memory that Direct3D has allocated to the cbuffer.

In summary - mapping structures from C# to HLSL cbuffers is non-trivial, but can certainly be done in entirely managed code. Have a look at the source code for DotWarp to see a complete working example.

6 comments

@lx
Mar 7, 2011
18:04

Prertty cool to see a SharpDX usage in action! :)

I have two remarks concerning the way to access a constant buffer in C#: 1) There is a performance issue in UpdateBufferValue. Allocating a DataStream each time you want to update a Constant buffer is usually a bad idea. You should better allocate this buffer once and somewhat attach it to the D3D11 Buffer (or create your own typed ConstantBuffer class that holds a preallocated constant buffer). Also, Marshal.StructureToPtr is usually very slow. If you don't have specific marshaling (unlike last sample using ByValArray) you could use SharpDX.Utilities.SizeOf() and dataStream.Write() which are much faster than StructureToPtr.

2) There is also a different approach to solve your problem (how to access and update transparently a constant buffer). You can make somekind of reflection on the constant buffer layout (shaderBytecode.Reflect()), query all variables and get their direct position in bytes in the constant buffer. You then declare an interface that mimics the constant buffers variable with properties. From this interface you can dynamically generate a proxy (using dynamic proxy or IL.Emit) that will use HLSL bytecode reflection to write directly data to the correct position.

I tend to prefer the second approach. It is asking a bit more work, but is more elegant and robust, as you don't have to know anything about HLSL fields alignment, you can even have different interfaces on the same constant buffers...etc.

Tim Jones
Mar 8, 2011
05:52

Thanks :)

I have updated my post using the suggestions from your first point. Since my app isn't real-time, I hadn't noticed the DataStream performance issue.

The second point is interesting - if I had a larger app, I'd probably use that. For my current app, the code above is pretty much all the marshalling I need to do, so I'll stick with hard-coded values.

Tim Jones
Mar 8, 2011
05:59

I forgot to say, I am using ShaderReflection for the unit tests I mentioned, to verify that I got the field offsets right. In case that code is useful to anybody, here it is:

private static void AssertFieldOffset(string name, ConstantBuffer constantBuffer) { int hlslOffset = constantBuffer.GetVariable(name).Description.StartOffset; Assert.That(Marshal.OffsetOf(typeof(T), name), Is.EqualTo(new IntPtr(hlslOffset))); }

ShaderReflection reflection = new ShaderReflection(bytecode); ConstantBuffer constantBuffer = reflection.GetConstantBuffer(1);

Assert.That(constantBuffer.Description.Size, Is.EqualTo(Marshal.SizeOf(typeof(BasicEffect.BasicEffectLightConstants)))); AssertFieldOffset("ActiveDirectionalLights", lightConstantBuffer); // etc... for other fields

mikerosss
May 4, 2011
07:47

Do you people have a facebook fan page? I looked for one on twitter but could not discover one, I would really like to become a fan!

Mike Pelton
Oct 29, 2011
11:49

Tim - hi - great piece and a life saver. One interesting wrinkle I spotted is that your UpdateValue method might be over-complicated - I'm passing the C# structure directly (attributed as you suggest - for me it's 3 matrices) so the only line in my version of UpdateValue is: _device.ImmediateContext.UpdateSubresource(ref value, _buffer, 0); ..which seems to work. This could, of course, be a happy coincidence, but it's definitely working to pass world, view and proj matrices. Will try and see if I can pass an array of lights later on, but can you see an obvious flaw in this approach?

Mike Pelton
Nov 5, 2011
17:50

That'll teach me! You are, of course, absolutely correct here - where we need proper marshalling the only safe way is to go the... Marshal.StructureToPtr(value, _dataStream.DataPointer, false); var dataBox = new DataBox(_dataStream.DataPointer); _device.ImmediateContext.UpdateSubresource(dataBox, _buffer, 0); ...route. I remain indebted to you!

Make a comment

Sorry, commenting has been temporary disabled because of spam. If you have any questions, you can email me, and you can also find me on Twitter.