Fred's continuous image representations

I am working on building continuous¹ representations of images in the context of special effects for cinema. I thus need to be able to re-synthesise an image from its representation at any resolution and possibly introducing some controlled modifications. To be insensitive to pixelisation, I extract the object boundaries at sub-pixel accuracy.

Briefly, I proceed in several steps:

• A segmentation (using relaxation labelling based on specified average colour and colour variance) gives regions at pixel accuracy, as well as probabilities for each pixel to belong to each region.
• The pixel-accurate regions are used to initialise snakes and the probabilities are used to make the snakes move towards the "true" sub-pixel boundaries.
• Finally, each object defined by a snake is represented. Currently, I have two types of representations. One is flat, i.e. an object is represented by its boundary (a NURBS curve) and a single colour (the average of the corresponding region). The second is smooth. In that case, an object is still represented by its boundary and by a triangulation whose vertices have a colour taken from the original image. This allows a smooth rendering of the object using a standard mesh renderer such as OpenGL^(TM). Other schemes are currently under investigation. One of these is based on texture analysis/representation/synthesis (see below).

The only user interaction required is the specification of parts of the image that are characteristic of the regions that the segmentation must produce. The segmentation is based on the average and the variance of the colour measured in each of the drawn region parts.

The following images show respectively an image of a goose, the image parts that are characteristic of the desired regions (drawn onto the original image), the regions produced by the relaxation labelling, the snakes after their convergence (drawn onto the original image), the mesh for the neck region, and the image rendered from the representation:

Other results can be found here while more details can be found in a technical report. A description of the synthesis/special effects aspects of this work can be found here.

The triangulation is done using a 2D quality mesh generator and Delaunay triangulator that ensures that generated triangles are not too large and do not have very acute angles. The size of the triangle must be of the order of the finest detail to be represented, which means that for a highly textured object, the representation can be very large in terms of number of primitives.

I am currently working on analysing textures to represent them in a continuous way to allow their re-synthesis also at any resolution.

Notes:

¹ By continuous, I mean vectorial, by opposition to discrete.