×

Other sites

Namespaces

Variants
Actions

Retro game console custom effect using Imaging SDK

From Nokia Developer Wiki
Jump to: navigation, search
Featured Article
27 Jul
2014

This article shows how to combine multiple image processing techniques like pixelation, colour quantization and dithering to simulate how a image would look in 8 and 16 bit videogame consoles.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code ExampleTested with
SDK: Windows Phone 8.0 SDK)
Devices(s): Nokia Lumia 920
Compatibility
Platform(s):
Windows Phone 8
Article
Created: r2d2rigo (14 Jul 2014)
Last edited: Loukt (29 Jul 2014)

Contents

Introduction

Today, photos and images come in high resolutions and with millions of colors. But there was a time where hardware wasn't powerful enough and each bit of data was expensive, so when making video games there were a lot of tricks involved to make the final result look beautiful enough. In this article, we are going to learn some of those tricks to process high-resolution, high-color images into downscaled, palette-reduced versions that would be able to be drawn according to the limitations of different 8 and 16-bit video game console hardware.

Techniques used

Cropping

Since the filter will be used to process different sizes of images, we are going to crop the processed area to a size multiple of the target hardware's resolution. This will leave some too wide or too tall images with a letterbox effect.

Pixelation

Instead of reducing the final image to the target hardware's resolution, the result will be upscaled as much as possible without surpassing the image size. Nearest neighbor filtering will be used to achieve this effect.

Color quantization

To accommodate the available palette of the target hardware, color quantization will be performed to reduce the number of colors displayed by the image. We will use two approaches for this:

  • Palette-based quantization: A limited palette of colors is supplied and the algorithm will use the Euclidean distance between the original color and the new one to match the most similar.
  • Bit-depth quantization: The number of bits to use for each color component is specified and the algorithm will truncate the current color to match it. An optional parameter is available to limit the number of different colors used at once by the image. If this limitation is provided, the effect will need a second pass where it will select the most used colors and re-quantize the image.

Dithering

Floyd-Steinberg dithering will be used to propagate quantization errors to neighboring pixels. This method dramatically improves image quality when targeting reduced palettes.

Implementing the effect

Boilerplate code

We will start by explaining some custom classes needed for the implementation of the effect.

struct ColorDelta

This struct contains a signed byte for each color component, excluding alpha. We need the sign because we will be performing color subtractions to obtain the quantization error, and the standard Windows.UI.Color structure doesn't offer this. 

public struct ColorDelta
{
    public sbyte R;
    public sbyte G;
    public sbyte B;
 
    public static ColorDelta operator +(ColorDelta c1, ColorDelta c2)
    {
        ColorDelta newDelta = c1.SafeAdd(c2);
 
        return newDelta;
    }
 
    public static ColorDelta operator *(ColorDelta c, float f)
    {
        ColorDelta newDelta = new ColorDelta();
        newDelta.R = (sbyte)(c.R * f);
        newDelta.R = (sbyte)(c.G * f);
        newDelta.R = (sbyte)(c.B * f);
 
        return newDelta;
    }
}

class Extensions

Extension methods to safely allow us to add together two existing Windows.UI.Color or ColorDelta taking into account under and overflow. 

public static class Extensions
{
    public static ColorDelta SafeAdd(this ColorDelta c1, ColorDelta c2)
    {
        ColorDelta newDelta = new ColorDelta();
        newDelta.R = (sbyte)ClampAdd(c1.R, c2.R, sbyte.MinValue, sbyte.MaxValue);
        newDelta.G = (sbyte)ClampAdd(c1.G, c2.G, sbyte.MinValue, sbyte.MaxValue);
        newDelta.B = (sbyte)ClampAdd(c1.B, c2.B, sbyte.MinValue, sbyte.MaxValue);
 
        return newDelta;
    }
 
    public static Color SafeAdd(this Color c1, ColorDelta c2)
    {
        Color newColor = new Color();
        newColor.R = (byte)ClampAdd(c1.R, c2.R, byte.MinValue, byte.MaxValue);
        newColor.G = (byte)ClampAdd(c1.G, c2.G, byte.MinValue, byte.MaxValue);
        newColor.B = (byte)ClampAdd(c1.B, c2.B, byte.MinValue, byte.MaxValue);
 
        return newColor;
    }
 
    private static int ClampAdd(int v1, int v2, int min, int max)
    {
        int result = v1 + v2;
 
        if (result > max)
        {
            return max;
        }
        else if (result < min)
        {
            return min;
        }
 
        return result;
    }
}

class RetroGameFilterSettingsBase

Base class for our two image filtering approaches. It stores a Windows.Foundation.Size for the target hardware's resolution and the abstract function Windows.UI.Color GetNearestColor(Windows.UI.Color color). This will allow us to create specific implementations for the color quantization process. 

public abstract class RetroGameFilterSettingsBase
{
    public Size Resolution { get; private set; }
 
    protected RetroGameFilterSettingsBase(Size resolution)
    {
        this.Resolution = resolution;
    }
 
    public abstract Color GetNearestColor(Color color);
}

Color quantization implementations

Palette-based

We will create a new class called RetroGameFilterPaletteSettings that inherits from RetroGameFilterSettingsBase. It will accept a new parameter in its constructor, a System.Collections.Generic.List<Windows.UI.Color> that holds all the valid palette entries.

The implementation of its Windows.UI.Color GetNearestColor(Windows.UI.Color color) function is as we described:

  • Initialize the minimum color distance to int.MaxValue.
  • For each palette entry
    • Check the Euclidean distance between the original color and the palette color.
    • If the distance is less than the minimum one already stored,
      • If the distance is 0 (exact match), return the palette color.
      • If not, store this color as the closest candidate and update the minimum color distance.
  • If no exact match is found, return the closest one. 
public class RetroGameFilterPaletteSettings : RetroGameFilterSettingsBase
{
    public List<Color> Palette { get; private set; }
 
    public RetroGameFilterPaletteSettings(Size resolution, List<Color> palette)
        : base(resolution)
    {
        this.Palette = palette;
    }
 
    public override Color GetNearestColor(Color color)
    {
        Color nearestColor;
        int minDelta = int.MaxValue;
 
        for (int i = 0; i < this.Palette.Count; ++i)
        {
            Color currentColor = this.Palette[i];
 
            int deltaR = color.R - currentColor.R;
            int deltaG = color.G - currentColor.G;
            int deltaB = color.B - currentColor.B;
 
            int totalDelta = ((deltaR * deltaR) + (deltaG * deltaG) + (deltaB * deltaB));
 
            if (totalDelta < minDelta)
            {
                if (totalDelta == 0)
                {
                    return currentColor;
                }
 
                minDelta = totalDelta;
                nearestColor = currentColor;
            }
        }
 
        return nearestColor;
    }
}

Bit-depth based

For this approach we will create another new class that inherits from RetroGameFilterSettingsBase too, called RetroGameFilterBitDepthSettings. It will receive three additional byte parameters in its constructor to specify the color depth of each RGB component, and an optional int one to limit the amount of active palette colors.

Its Windows.UI.Color GetNearestColor(Windows.UI.Color color) implementation is very simple:

  • For each colour component,
    • Calculate a bit mask by shifting the 0xFF value to the left as many times as the difference between the maximum bit depth (8) and the component's one.
    • Perform a binary and (&) between the color component and the calculated mask to truncate the desired amount of least significant bits.
  • Return a new color with the quantized values and full alpha. 
public class RetroGameFilterBitDepthSettings : RetroGameFilterSettingsBase
{
    public byte RedDepth { get; private set; }
    public byte GreenDepth { get; private set; }
    public byte BlueDepth { get; private set; }
    public int PaletteLimit { get; private set; }
 
    public RetroGameFilterBitDepthSettings(Size resolution, byte redDepth, byte greenDepth, byte blueDepth, int paletteLimit = 0)
        : base(resolution)
    {
        if (redDepth > 8 || greenDepth > 8 || blueDepth > 8)
        {
            throw new InvalidOperationException("The maximum color depth for any component is 8 bits.");
        }
 
        this.RedDepth = redDepth;
        this.GreenDepth = greenDepth;
        this.BlueDepth = blueDepth;
        this.PaletteLimit = paletteLimit;
    }
 
    public override Color GetNearestColor(Color color)
    {
        Color newColor = new Color();
        newColor.A = 0xff;
        newColor.R = (byte)(color.R & (0xff << (8 - this.RedDepth)));
        newColor.G = (byte)(color.G & (0xff << (8 - this.GreenDepth)));
        newColor.B = (byte)(color.B & (0xff << (8 - this.BlueDepth)));
 
        return newColor;
    }
}

Custom effect implementation

Since we will need to process entire image rows at once for our dithering algorithm to work, we need to base the effect off the Nokia.Graphics.Imaging.CustomEffectBase class. This will allow us to process the image from left to right, top to bottom and correctly propagate the quantization error values to the right and bottom. Constant float values for the error's distribution will be defined according to the algorithm's requirements:

... * 7.0f / 16.0f
3.0f / 16.0f 5.0f / 16.0f 1.0f / 16.0f

The asterisk (*) denotes the currently processed pixel. Error values are spread according to the multiplier values of adjacent pixels.

We will declare different ColorDelta and ColorDelta[] private properties to hold the error values of future pixels. Also, a number of predefined configurations have been defined as an example, as will be detailed in the next section.

Predefined settings

A number of predefined settings have been added to the filter as static read-only fields. These encapsulate the settings for some popular 8 and 16-bit videogame consoles, which are:

  • Nintendo Game Boy: 160x144 pixels[1], 4-color palette. The palette has been adjusted so instead of greys it uses the greenish tint of the original LCD display[2].
  • Nintendo Game Boy Color: 160x144 pixels[3], 15-bit RGB palette with a maximum of 56 simultaneous colours[4].
  • Nintendo NES: 256x240 pixels[5] with a palette of 54 different colours[6].
  • Nintendo SNES: 256x224 pixels[7], 15-bit RGB palette with a maximum of 256 simultaneous colours[8].
  • Sega Master System: 256x240 pixels[9], 6-bit RGB palette with a maximum of 32 simultaneous colours[10].
  • Sega Mega Drive: 320x240 pixels[11], 9-bit RGB palette with a maximum of 61 simultaneous colours[12].

Although while trying to be as accurate possible to the original platform limitations, some specific ones have been ignored in favor of simplified code. For resolution, PAL video mode has been selected instead of NTSC.

Helper functions

Before moving to the core of the OnProcess function, we will detail two additional functions that help make the code cleaner:

AdvanceErrorRow

This simple function is called when a new image row is going to be processed and resets the appropriate accumulated error values. 

private void AdvanceErrorRow()
{
    int intResX = (int)this.filterSettings.Resolution.Width;
 
    this.nextPixelError = new ColorDelta();
    this.currentRowError = nextRowError;
    this.nextRowError = new ColorDelta[intResX];
}
QuantizeColor

Returns the quantized value of a specific pixel according to the current RetroGameFilterSettingsBase used by the filter. It also accumulates the error values using the dithering algorithm. 

private Color QuantizeColor(Color pixelColor, int pixelX)
{
    ColorDelta totalError = this.nextPixelError + this.currentRowError[pixelX];
    pixelColor = pixelColor.SafeAdd(totalError);
 
    Color nearestColor = this.filterSettings.GetNearestColor(pixelColor);
 
    ColorDelta error = new ColorDelta();
    error.R = (sbyte)(pixelColor.R - nearestColor.R);
    error.G = (sbyte)(pixelColor.G - nearestColor.G);
    error.B = (sbyte)(pixelColor.B - nearestColor.B);
 
    if (pixelX > 0)
    {
        nextRowError[pixelX - 1] += error * DOWN_LEFT_PIXEL_ERROR;
    }
 
    if (pixelX < this.filterSettings.Resolution.Width - 1)
    {
        nextPixelError = error * RIGHT_PIXEL_ERROR;
 
        nextRowError[pixelX + 1] += error * DOWN_RIGHT_PIXEL_ERROR;
    }
 
    nextRowError[pixelX] += error * DOWN_PIXEL_ERROR;
 
    return nearestColor;
}

Image processing

Finally we arrive to the OnProcess function, which will take the input image and output a new one according to the hardware restrictions we specify. The flow is as follows:

  • Check that source image is big enough (at least the size of the target hardware resolution).
  • Get a scaled pixel size that fits in the output image and create a cropping area with a size multiple of the target resolution.
  • Create a color map array with the same size as the target resolution.
  • Store the need for a second pass in a boolean variable. Only RetroGameFilterBitDepthSettings with a specified palette limit need a second pass.
  • For each image row,
    • Check that pixel to be processed lies inside cropping region. If not, jump to the next iteration.
    • If the color map value that this pixel needs to use hasn't been initialized yet,
      • Obtain the pixel color via nearest neighbor filtering.
      • Quantize the color and store it in the color map.
      • If we need a second pass, a System.Collections.Generic.Dictionary<Windows.UI.Color, int> will be needed to store how many times each quantized color appears in the transformed image. Create or update the value inside the dictionary.
    • If we don't need a second pass, write the quantized color to the output image.
  • If a second pass is needed,
    • Sort the dictionary by dominant colours first and take the first N values, where N is the palette limit of the specified RetroGameFilterBitDepthSettings. Set the filterSettings of the filter to a new instance of RetroGameFilterPaletteSettings, with the same resolution and specifying the new color palette.
    • Quantize again the color map according to the reduced palette.
    • For each image row,
    • Check that pixel to be processed lies inside cropping region. If not, jump to the next iteration.
    • Obtain the quantized color for this pixel's position from the color map and write it to the output image. 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
    Size size = sourcePixelRegion.ImageSize;
 
    if (size.Width < this.filterSettings.Resolution.Width ||
        size.Height < this.filterSettings.Resolution.Height)
    {
        throw new InvalidOperationException("Target image is too small!");
    }
 
    int pixelSize;
    Rect cropArea = new Rect();
 
    if (size.Width > size.Height)
    {
        pixelSize = (int)(size.Height / this.filterSettings.Resolution.Height);
    }
    else
    {
        pixelSize = (int)(size.Width / this.filterSettings.Resolution.Width);
    }
 
    cropArea.Width = pixelSize * this.filterSettings.Resolution.Width;
    cropArea.Height = pixelSize * this.filterSettings.Resolution.Height;
    cropArea.X = (size.Width - cropArea.Width) / 2.0f;
    cropArea.Y = (size.Height - cropArea.Height) / 2.0f;
 
    uint[] sourcePixels = sourcePixelRegion.ImagePixels;
    uint[] targetPixels = targetPixelRegion.ImagePixels;
 
    int intResX = (int)this.filterSettings.Resolution.Width;
    int intResY = (int)this.filterSettings.Resolution.Height;
    uint[,] colorMap = new uint[intResX, intResY];
 
    this.nextRowError = new ColorDelta[intResX];
 
    Dictionary<Color, int> imagePalette = new Dictionary<Color, int>();
    bool needsSecondPass = this.filterSettings is RetroGameFilterBitDepthSettings && (this.filterSettings as RetroGameFilterBitDepthSettings).PaletteLimit > 0;
 
    sourcePixelRegion.ForEachRow((index, width, startPosition) =>
    {
        int y = (int)startPosition.Y;
 
        if (y < cropArea.Y || y >= (cropArea.Y + cropArea.Height))
        {
            return;
        }
 
        this.AdvanceErrorRow();
 
        index += (int)cropArea.X;
        for (int x = (int)cropArea.X; x < (cropArea.X + cropArea.Width); ++x, ++index)
        {
            int mapX = (int)((x - cropArea.X) / pixelSize);
            int mapY = (int)((y - cropArea.Y) / pixelSize);
 
            if (colorMap[mapX, mapY] == 0)
            {
                uint pixel = sourcePixels[index + (int)(pixelSize * 0.5f) + sourcePixelRegion.Pitch * (int)(pixelSize * 0.5f)];
 
                Color pixelColor = ToColor(pixel);
                Color nearestColor = this.QuantizeColor(pixelColor, mapX);
 
                if (needsSecondPass)
                {
                    if (!imagePalette.ContainsKey(nearestColor))
                    {
                        imagePalette.Add(nearestColor, 0);
                    }
                    else
                    {
                        imagePalette[nearestColor] += 1;
                    }
                }
 
                colorMap[mapX, mapY] = (uint)(0xff000000 | (nearestColor.R << 16) | (nearestColor.G << 8) | nearestColor.B);
            }
 
            if (!needsSecondPass)
            {
                targetPixels[index] = colorMap[mapX, mapY];
            }
        }
    });
 
    if (needsSecondPass)
    {
        int paletteLimit = (this.filterSettings as RetroGameFilterBitDepthSettings).PaletteLimit;
        List<Color> reducedPalette = imagePalette.OrderBy(entry => entry.Value).Select(entry => entry.Key).Take(paletteLimit).ToList();
        this.filterSettings = new RetroGameFilterPaletteSettings(this.filterSettings.Resolution, reducedPalette);
 
        nextRowError = new ColorDelta[intResX];
 
        for (int y = 0; y < this.filterSettings.Resolution.Height; ++y)
        {
            this.AdvanceErrorRow();
 
            for (int x = 0; x < this.filterSettings.Resolution.Width; ++x)
            {
                uint pixel = colorMap[x, y];
                Color pixelColor = ToColor(pixel);
                Color nearestColor = this.QuantizeColor(pixelColor, x);
 
                colorMap[x, y] = (uint)(0xff000000 | (nearestColor.R << 16) | (nearestColor.G << 8) | nearestColor.B);
            }
        }
 
        sourcePixelRegion.ForEachRow((index, width, startPosition) =>
        {
            int y = (int)startPosition.Y;
 
            if (y < cropArea.Y || y >= (cropArea.Y + cropArea.Height))
            {
                return;
            }
 
            index += (int)cropArea.X;
            for (int x = (int)cropArea.X; x < (cropArea.X + cropArea.Width); ++x, ++index)
            {
                int mapX = (int)((x - cropArea.X) / pixelSize);
                int mapY = (int)((y - cropArea.Y) / pixelSize);
 
                targetPixels[index] = colorMap[mapX, mapY];
            }
        });
    }
}

Using the filter

Assuming basic knowledge of the Nokia Imaging SDK, the custom effect is ready to be used. You just need to instantiate it passing an IImageProvider and a processing setting of your choice or an already existing one, and display the results into an Image control or use it as you wish. 

// The already loaded source image to be processed.
StreamImageSource imageSource;
// Image control where the result will be displayed.
Image outputImage;
// Output bitmap with the processed image.
WriteableBitmap outputBitmap;
// Bitmap buffer used for rendering the image.
WriteableBitmap writeableBitmap = new WriteableBitmap((int)outputImage.Width, (int)outputImage.Height);
 
// Instantiate the filter with a predefined setting.
using (RetroGameFilter filter = new RetroGameFilter(imageSource, RetroGameFilter.GameBoyColorSettings))
{
    // Create a bitmap renderer.
    using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(filter, writeableBitmap))
    {
        // Render the effect and retrieve the output bitmap.
        outputBitmap = await renderer.RenderAsync();
    }
}
 
// Set the source of the Image control to the processed bitmap.
outputImage.Source = outputBitmap;

Remarks

Although some optimizations have been done to the filter, it is too slow to use on real-time applications in some cases. While configurations like the predefined Game Boy or Game Boy Color can process a 1024x768 image in less than a second on a Lumia 920, others like the Super Nintendo (which needs two passes) take almost 5 seconds. Using unsafe code, or better, native code with ARM instructions could speed the process to acceptable times.

A commented version of the full source code for the effect can be found attached to this article.

Sample output

Below is a gallery of sample images and how they appear after applying each one of the predefined settings.

Hardware "Shoes" sample image "Five" sample image "Pool" sample image
Original image
Zero.jpg
Five.jpg
Six.jpg
Game Boy
Zero gameboy.jpg
Five gameboy.jpg
Six gameboy.jpg
Game Boy Color
Zero gameboycolor.jpg
Five gameboycolor.jpg
Six gameboycolor.jpg
NES
Zero nes.jpg
Five nes.jpg
Six nes.jpg
SNES
Zero snes.jpg
Five snes.jpg
Six snes.jpg
Master System
Zero mastersystem.jpg
Five mastersystem.jpg
Six mastersystem.jpg
Mega Drive
Zero megadrive.jpg
Five megadrive.jpg
Six megadrive.jpg

From these images, we can infer some conclusions:

  • Images with very close colors, like "Pool" sample, don't look well with limited palettes even with dithering.
  • Images with high contrast and small color variation, like "Five" sample, get the best results on all platforms.
  • Mega Drive has the highest resolution, but with such a low palette limit image colors don't match well.
  • Super Nintendo is quite expensive to compute with the high limit of 256 palette colors, but it has the best overall image quality of all configurations.

References

  1. ↑ http://en.wikipedia.org/wiki/Game_boy#Technical_specifications
  2. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#Original_Game_Boy
  3. ↑ http://en.wikipedia.org/wiki/Game_Boy_Color#Summary
  4. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#Game_Boy_Color
  5. ↑ http://en.wikipedia.org/wiki/Nintendo_Entertainment_System_technical_specifications#Video
  6. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#Famicom.2FNES
  7. ↑ http://en.wikipedia.org/wiki/Snes#Video
  8. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#SuperFamicom.2FSNES
  9. ↑ http://en.wikipedia.org/wiki/Master_system#Technical_specifications
  10. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#Master_System
  11. ↑ http://en.wikipedia.org/wiki/Mega_Drive#Technical_specifications
  12. ↑ http://en.wikipedia.org/wiki/List_of_videogame_console_palettes#Mega_Drive.2FGenesis
Retrieved from "http://developer.nokia.com/community/wiki/index.php?title=Retro_game_console_custom_effect_using_Imaging_SDK&oldid=221737"
This page was last modified on 29 July 2014, at 02:49.
279 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×