Skip Navigation LinksHome > View Post

Generating images using WPF on the Server

… and ‘plugging any nasty leaks you might see’.

Recently I worked on a proof of concept where we wanted to leverage the power of WPF and DataTemplates on a server to generate images. This is actually surprisingly easy and, unsurprisingly, very powerful.

The project planned to use WPF to generate images for a PivotViewer collection. If you haven’t checked out PivotViewer yet you really should – it even comes in Silverlight form these days and it’s all kinds of great. A phenomenal way to add simple BI like slicing and dicing of data with mind-blowing visuals and transitions.

To enhance the raw visuals in the source data (just portraits of people in this case) we wanted to generate new source images with a data overlay. Something like this:

image

With a mugshot at the back and some overlaid visuals to create a composite image. In this example we simply have a simple KPI indicator and the name of the person.  A bit of colour adds an extra visual dimension to the PivotViewer and can be used to great effect.

image

Of course, the idea of overlaying the PivotViewer source images is such a good idea that the Pivot team already thought of it. If you use their PAuthor.exe tool then you can specify a HTML template to be used for an overlay. However, in this exercise we had a dynamic collection with a dynamic collection of images and wouldn’t be using the command line tool. Also, with the developers familiar with Silverlight it made sense to opt for the infinitely more powerful WPF/DataTemplate approach.

The DataTemplate would perhaps look something like this (btw, this is a different template to that shown above):

<DataTemplate 
xmlns
="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
    <Grid Background="White" Height="500" Width="350">
        <Viewbox>
            <Image Source="{Binding ImageSource}"/>
        </Viewbox>
        <StackPanel Margin="35">
            <TextBlock
Text
="{Binding Attributes[Name], StringFormat='Name: \{0\}'}"
FontSize
="20" FontWeight="Bold"/>
            <TextBlock
Text
="{Binding Attributes[HireDate], StringFormat='Hired: \{0:d\}'}"
FontSize
="20" />
            <TextBlock
Text
="{Binding Attributes[TargetPercentage], StringFormat='\{0\}'}"
FontSize
="20" />
        </StackPanel>
        <Border Name="TargetBorder" BorderBrush="Red"
BorderThickness
="30" Opacity=".5"/>
    </Grid>

    <DataTemplate.Triggers>
        <DataTrigger Binding="{Binding Attributes[OnTarget]}"
Value
="True">
            <Setter TargetName="TargetBorder"
Property
="BorderBrush" Value="Green" />
        </DataTrigger>
    </DataTemplate.Triggers>

</DataTemplate>

Notice how it makes heavy use of data binding and even a DataTrigger that would switch the border color from Red to Green if the OnTarget value is true. This DataTemplate would need some DataContext object it would be bound to, which might look like this (notice, this is a one-time only binding so no need for INotifyPropertyChanged here).

public class TemplateBindingSource
{
    public ImageSource ImageSource { get; set; }
    public Dictionary<string, object> Attributes { get; set; }
}

You’ve probably already guessed that this little chap will carry the ‘mugshot’ image (in-memory) and the Attributes dictionary is a simple name-value pairing of keys and data. You could easily imagine inflating this dictionary from almost any data source (an object via reflection, or a row of data from a database with each column representing a key).

The meat of the processing takes place in this method:

public byte[] ComposeImage(
Dictionary<string, object> attributes,
byte[] imageData)
{
    byte[] result = null;

    _dispatcher.Invoke((Action)delegate
        {
            using (MemoryStream stream = new MemoryStream(imageData))
            {
                //// Load the bitmap image from image bytes
                BitmapImage image = new BitmapImage();

                image.CacheOption = BitmapCacheOption.None;
                image.BeginInit();
                image.StreamSource = stream;
                image.EndInit();
                image.Freeze();

                // Create the data context that the template will use
                TemplateBindingSource dc = new TemplateBindingSource
                {
                    ImageSource = image,
                    Attributes = attributes
                };

                DataTemplate template = TemplateService.GetTemplate();

                ContentControl element = new ContentControl
                {
                    ContentTemplate = template,
                    Content = dc
                };

                // Measure and arrange the tile
                element.Measure(new Size {
Height = double.PositiveInfinity,
Width = double.PositiveInfinity });
                element.Arrange(new Rect(0, 0,
element.DesiredSize.Width,
element.DesiredSize.Height));

                element.UpdateLayout();

                result = WriteVisualAsJpeg(element);
            }
        });

    return result;
}

The TemplateService.GetTemplate() call simply loads the DataTemplate into memory from disk, something like this:

using (Stream steam = File.OpenRead(filename))
{
    return (DataTemplate) XamlReader.Load(stream);
}

Of course, if you’re processing a lot of images then you wouldn’t want to read and parse the DataTemplate xaml file repeatedly. You’d want to cache this somewhere (be careful here, only the thread that created the DataTemplate can play with it!). The other interesting methods involved in this process are CopyStream (taken from this post) and the genuinely interesting WriteVisualAsJpeg:

private byte[] WriteVisualAsJpeg(FrameworkElement element)
{
    double factor = 1d;

    int w = Convert.ToInt32(element.ActualWidth);
    int h = Convert.ToInt32(element.ActualHeight);

    RenderTargetBitmap render = new RenderTargetBitmap(
w, h, 96 * factor, 96 * factor, PixelFormats.Default);
    render.Render(element);

    BitmapEncoder encoder = new JpegBitmapEncoder();
    encoder.Frames.Add(BitmapFrame.Create(render));

    using (MemoryStream stream = new MemoryStream())
    {
        encoder.Save(stream);

        return stream.GetBuffer();
    }
}

To write the result to disk you can just File.WriteAllBytes() with the result of ComposeImage() above. Lemon Squeezy.

When I’m leaking images…

However, we did see one really notable problem with our first shot at this technique…

This was the memory footprint of our application over time:

clip_image002

Which resulted in this:

clip_image001

Ouch! Yup, an ever increasing amount of memory followed by an OutOfMemoryException. A sure sign that we’re haemorrhaging memory somewhere.

With some help of the smart folk on the WPF team (thanks guys!) we determined that the problem lay with the fact that the WPF code was running on a thread without an active Dispatcher to process any queued frames and we had been missing a key call to update the layout of the element. This was fixed by, amusingly enough, introducing a running Dispatcher the the equation and calling element.UpdateLayout() as highlighted above.

The bigger problem of course is running a Dispatcher on the server – not to mention that we need an STA thread to do any WPF work at all. Enter the BackgroundStaDispatcher class:

public class BackgroundStaDispatcher
{
    private Dispatcher _dispatcher;


    public BackgroundStaDispatcher(string name, TraceSource logger)
    {
        AutoResetEvent are = new AutoResetEvent(false);

        Thread thread = new Thread((ThreadStart)delegate
        {
            _dispatcher = Dispatcher.CurrentDispatcher;
            _dispatcher.UnhandledException +=
delegate(
object sender,
DispatcherUnhandledExceptionEventArgs e)
            {
                logger.TraceData(TraceEventType.Error, 0, e.Exception);
                if (!Debugger.IsAttached)
                {
                    e.Handled = true;
                }
            };
            are.Set();
            Dispatcher.Run();
        });

        thread.Name = string.Format("BackgroundStaDispatcher({0})", name);
        thread.SetApartmentState(ApartmentState.STA);
        thread.IsBackground = true;
        thread.Start();

        are.WaitOne();
    }


    public void Invoke(Action action)
    {
        _dispatcher.Invoke(action);
    }


    public void BeginInvoke(Action action)
    {
        _dispatcher.BeginInvoke(action);
    }
}

This class just creates a new STA Thread and gets a Dispatcher running on it. As you can see in the highlighted section in ComposeImage above (_dispatcher.Invoke) we’re using an instance of this guy to do all of the WPF grunt work. With this, and the call to element.UpdateLayout our memory profile looks like this:

clip_image002[13]

Much better!

Tags: MVVM WPF

 
Josh Post By Josh Twist
4:58 PM
11 Oct 2010

» Next Post: Getting Good at Parallel (with NxtGenUG)
« Previous Post: Cloud Artwork

Comments are closed for this post.

Posted by Christo @ 12 Oct 2010 8:51 AM
Then this must be the solution to:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=489723&wa=wsignin1.0#

We've had this memory bug now for so long, but couldn't find anyone or any solution which would shed some light on this bug.

Thanks!

Posted by Josh @ 12 Oct 2010 8:54 AM
That's good news - glad the post helped!

Posted by James World @ 13 Oct 2010 12:47 PM
That's done it for me. Morgan must surely now forever be known as Malcom Gladwell... very useful post, thanks Josh.

Posted by James World @ 13 Oct 2010 12:50 PM
U need an idempotent comment submission! ;) 3G shenanigans responsible here!

Posted by Christo @ 21 Oct 2010 7:19 AM
I've thought a little about the solution, but was wondering if you could find out why shutting down the dispatcher does not release the memory? Have a look at the sample in the connect issue to see what I'm referring to.

At least for now we're keeping our memory footprint down and within acceptable limits, but we're also routing all background rendering through one dispatcher. The only option we've got to increase our throughput would be to create a pool of background dispatchers.

Posted by Jiri @ 03 Nov 2010 12:19 PM
I tried your sample - I need generate picture on IIS. I think, there's missing line - _dispatcher.InvokeShutdown(). I think, InvokeShutdown clears thread properly. Without this line, every request created new thread, which stayed in web-server process.

Posted by Josh @ 03 Nov 2010 1:22 PM
@Jiri - In my sample we had a console application that terminated at the end of the processing - so this wasn't a problem. For IIS, rather than create a BackgroundStaDispatcher per request (I *really* wouldn't recommend this!) I would create a pool of them to be reused throughout the lifteime of the app domain. You could achieve this by using a ThreadLocal<BackgroundStaDispatcher> so there is one per ThreadPool thread automatically and it can be reused. As always test and test and test again!

@Christo - see above :) Aso, closing the dispatcher won't clear the memory. The dispatcher needs to be available to process queued work (caused by UpdateLayout) that removeds some rooted references.

Posted by Rich @ 14 Jan 2011 8:00 PM
I'm trying to do something similar but I'm having a hard time getting an image to generate in IIS7.5 running .NET 4.0. The code works fine in Cassini, but fails to generate an image in IIS (I just get a black image). Have you seen anything like that?

Posted by Glenn Slayden @ 03 Mar 2011 3:03 AM
@Rich, it's probably because you didn't set up a PresentationSource such as System.Windows.Interop.HwndSource. I just got server WPF rendering working on IIS7.5 and .NET 4.0 but it was all black before this. Check out http://stackoverflow.com/questions/2851236/rendertargetbitmap-resourced-visualbrush-incomplete-image for the solution.

© 2005 - 2014 Josh Twist - All Rights Reserved.