![[ImageMagick]](../images/script.png)
![[sponsor]](../images/networkredux.png)
The citizens of Oz were quite content with their benefactor, the all-powerful Wizard. They accepted his wisdom and benevolence without ever questioning the who, why, and where of his power. Like the citizens of Oz, if you feel comfortable that ImageMagick can help you convert, edit, or compose your images without knowing what goes on behind the curtain, feel free to skip this section. However, if you want to know more about the software and algorithms behind ImageMagick, read on. To fully benefit from this discussion, you should be comfortable with image nomenclature and be familiar with computer programming.
Architecture Overview
An image typically consists of a rectangular region of pixels and metadata. To convert, edit, or compose an image in an efficient manner we need convenient access to any pixel anywhere within the region (and sometimes outside the region). And in the case of an image sequence, we need access to any pixel of any region of any image in the sequence. However, there are hundreds of image formats such JPEG, TIFF, PNG, GIF, etc., that makes it difficult to access pixels on demand. Within these formats we find differences in:
- colorspace (e.g RGB, CMYK, YUV, Lab, etc.)
- bit depth (.e.g 1, 4, 8, 12, 16, etc.)
- storage format (e.g. unsigned, signed, float, double, etc.)
- compression (e.g. uncompressed, RLE, Zip, BZip, etc.)
- orientation (i.e. top-to-bottom, right-to-left, etc.),
- layout (.e.g. raw, interspersed with opcodes, etc.)
In addition, some image pixels may require attenuation, some formats permit more than one frame, and some formats contain vector graphics that must first be rasterized (converted from vector to pixels).
An efficient implementation of an image processing algorithm may require we get or set:
- one pixel a time (e.g. pixel at location 10,3)
- a single scanline (e.g. all pixels from row 4)
- a few scanlines at once (e.g. pixel rows 4-7)
- a single column or columns of pixels (e.g. all pixels from column 11)
- an arbitrary region of pixels from the image (e.g. pixels defined at 10,7 to 10,19)
- a pixel in random order (e.g. pixel at 14,15 and 640,480)
- pixels from two different images (e.g. pixel at 5,1 from image 1 and pixel at 5,1 from image 2)
- pixels outside the boundaries of the image (e.g. pixel at -1,-3)
- a pixel component that is unsigned or in a floating-point representation (e.g. 0.17836)
- a high-dynamic range pixel that can include negative values as well as values that exceed the quantum depth (e.g. -0.00716)
- one or more pixels simultaneously in different threads of execution
In addition, some images include a clip mask that define which pixels are eligible to be updated. Pixels outside the area defined by the clip mask remain untouched.
Given the varied image formats and image processing requirements, we implemented the ImageMagick pixel cache to provide convenient sequential or parallel access to any pixel on demand anywhere inside the image region and from any image in a sequence. In addition, the pixel cache permits access to pixels outside the boundaries defined by the image (we call these virtual pixels).
In addition to pixels, images have a plethora of image properties and profiles. Properties include the well known items such as width, height, depth, and colorspace. An image may have optional properties which might include the image author, a comment, a create date, and others. Some images also include profiles for color management, or EXIF, IPTC, 8BIM, or XMP informational profiles. ImageMagick provides command line options and programming methods to get, set, or view image properties or profiles or apply profiles.
ImageMagick consists of more than 375,000 lines of C code and optionally depends on several million lines of code in dependent libraries (e.g. JPEG, PNG, TIFF libraries). Given that, one might expect a huge architecture document. However, a great majority of image processing is simply accessing pixels and its metadata and our simple and elegant implementation makes this easy for the ImageMagick developer. We discuss the implementation of the pixel cache and getting and setting image properties and profiles in the next few sections. Next, we discuss using ImageMagick within a thread of execution. In the final sections, we discuss image coders to read or write a particular image format followed by a few words on creating a filter to access or update pixels based on your custom requirements.
The Pixel Cache
The ImageMagick pixel cache is a repository for image pixels with up to 5 channels. The first 4 channels are stored contiguously and an optional second area follows with 1 channel. The channels are at the depth specified when ImageMagick was built. The channel depths are 8 bits-per-pixel component for the Q8 version of ImageMagick, 16 bits-per-pixel component for the Q16 version, and 32 bits-per-pixel component for the Q32 version. By default pixel components are unsigned quantities, however, if you use the high dynamic-range version of ImageMagick, the components are 32-bit floating point. The primary 4 channels can hold any value but typically contain red, green, blue, and alpha intensities or cyan, magenta, yellow, and alpha intensities. The optional fifth channel contains the colormap indexes for colormapped images or the black channel for CMYK images. The pixel cache storage may be heap memory, anonymous memory mapped memory, disk-backed memory mapped, or on disk. The pixel cache is reference-counted. Only the cache properties are copied when the cache is cloned. The cache pixels are subsequently copied when you signal your intention to update any of the pixels.
Create the Pixel Cache
The pixel cache is associated with an image when it is created and it is initialized when you try to get or put pixels. Here are three common methods to associate a pixel cache with an image:
Create an image canvas initialized to the background color:
image=AllocateImage(image_info); if (SetImageExtent(image,640,480) == MagickFalse) { /* an exception was thrown */ } (void) QueryMagickColor("red",&image->background_color,&image->exception); SetImageBackgroundColor(image);
Create an image from a JPEG image on disk:
(void) strcpy(image_info->filename,"image.jpg"): image=ReadImage(image_info,exception); if (image == (Image *) NULL) { /* an exception was thrown */ }
Create an image from a memory based image:
image=BlobToImage(blob_info,blob,extent,exception); if (image == (Image *) NULL) { /* an exception was thrown */ }
In our discussion of the pixel cache we use the MagickCore API to illustrate our points, however, the principles are the same for other program interfaces to ImageMagick.
When the pixel cache is initialized, pixels are scaled from whatever bit depth they originated from to that required by the pixel cache. For example, a 1-channel 1-bit monochrome PBM image is scaled to a 4 channel 8-bit RGBA image, if you are using the Q8 version of ImageMagick, and 16-bit RGBA for the Q16 version. You can determine which version you have using the ‑version option, as with this command:
$magick> identify -versionVersion: ImageMagick 6.5.8-7 2009-12-05 Q16 http://www.imagemagick.org
As you can see, the convenience of the pixel cache sometimes comes with a trade-off in storage (e.g. storing a 1-bit monochrome image as 16-bit RGBA is wasteful) and speed (i.e. storing the entire image in memory is generally slower than accessing one scanline of pixels at a time).
Access the Pixel Cache
Once the pixel cache is associated with an image, you typically want to get, update, or put pixels into it. We refer to pixels inside the image region as authentic pixels and outside the region as virtual pixels. Use these methods to access the pixels in the cache:
- GetVirtualPixels() gets pixels that you do not intend to modify
- GetAuthenticPixels() gets pixels that you intend to modify
- QueueAuthenticPixels() queue pixels that you intend to modify
- SyncAuthenticPixels() update the pixel cache with any modified pixels
Here is a typical MagickCore code snippet for manipulating pixels in the pixel cache. In our example we copy pixels from the input image to the output image and decrease the intensity by 10%:
When we first create the destination image by cloning the source image, the pixel cache pixels are not copied. They are only copied when you signal your intentions to modify the pixel cache by calling GetAuthenticPixels() or QueueAuthenticPixels(). Use QueueAuthenticPixels() if you want to set new pixel values rather than update existing ones. Finally, use SyncAuthenticPixels() to ensure any updated pixels are pushed to the pixel cache.
Recall how we mentioned that the indexes of a colormapped image or the black channel of a CMYK image are stored separately. Use GetVirtualIndexes() (to read the indexes) or GetAuthenticIndexes() (to update the indexes) to gain access to this channel. For example, to print the colormap indexes, use:
const IndexPacket
*indexes;
for (y=0; y < (long) source->rows; y++)
{
p=GetVirtualPixels(source,0,y,source->columns,1);
if (p == (const PixelPacket *) NULL)
break;
indexes=GetVirtualIndexes(source);
for (x=0; x < (long) source->columns; x++)
(void) printf("%d\n",indexes[x];
}
if (y < (long) source->rows)
/* an exception was thrown */
The pixel cache manager decides whether to give you direct or indirect access to the image pixels. In some cases the pixels are staged to an intermediate buffer-- and that is why you must call SyncAuthenticPixels() to ensure this buffer is pushed out to the pixel cache to guarantee the corresponding pixels in the cache are updated. For this reason we recommend that you only read or update a scanline or a few scanlines of pixels at a time. However, you can get any rectangular region of pixels you want. GetAuthenticPixels() requires that the region you request is within the bounds of the image area. For a 640 by 480 image, you can get a scanline of 640 pixels but if you ask for 641 pixels, an exception is returned. GetVirtualPixels() does not have this constraint. For example,
p=GetVirtualPixels(source,-3,3,source->columns+7,7,exception);
gives you the pixels you asked for without complaint, even though some are not within the confines of the image region.
Virtual Pixels
Access to the virtual pixels are controlled by the SetImageVirtualPixelMethod() method from the MagickCore API or the ‑virtual‑pixel option from the command line. The methods include:
background: the area surrounding the image is the background color black: the area surrounding the image is black checker-tile: alternate squares with image and background color dither: non-random 32x32 dithered pattern edge: extend the edge pixel toward infinity gray: the area surrounding the image is gray horizontal-tile: horizontally tile the image, background color above/below horizontal-tile-edge: horizontally tile the image and replicate the side edge pixels mirror: mirror tile the image random: choose a random pixel from the image tile: tile the image (default) transparent: the area surrounding the image is transparent blackness vertical-tile: vertically tile the image, sides are background color vertical-tile-edge: vertically tile the image and replicate the side edge pixels white: the area surrounding the image is white
There is a plethora of image processing algorithms that require a neighborhood of pixels about a pixel of interest. There is typically a caveat concerning how to handle pixels around the image boundaries, known as edge pixels. With virtual pixels, you do not need to concern yourself about special edge processing other than choosing which virtual pixel method is most appropriate for your algorithm.
Cache Storage and Resource Requirements
We mentioned previously that this simple and elegant design of the ImageMagick pixel cache comes at a cost in terms of storage and processing speed. The pixel cache storage requirements scales with the area of the image and the bit depth of the pixel components. For example, if we have a 640 by 480 image and we're using the Q16 version of ImageMagick, the pixel cache consumes image width * height * bit-depth / 8 * channels bytes or approximately 2.3 mebibytes (i.e. 640 * 480 * 2 * 4). Not too bad, but what if your image is 25000 by 25000 pixels? The pixel cache requires approximately 4.7 gibibytes of storage. Ouch. ImageMagick accounts for possible huge storage requirements by caching large images to disk rather than memory. Typically the pixel cache is stored in memory using heap memory. If heap memory is exhausted, pixels are stored in in an anonymous map; if the anonymous memory map is exhausted, we create the pixel cache on disk and attempt to memory-map it; and if memory-map memory is exhausted, we simply use standard disk I/O. Disk storage is cheap but it is also very slow, upwards of 1000 times slower than memory. We can get some speed improvements, up to 5 times, if we use memory mapping to the disk-based cache. These decisions about storage are made automagically by the pixel cache manager negotiating with the operating system. However, you can influence how the pixel cache manager allocates the pixel cache with cache resource limits. The limits include:
- files
- maximum number of open pixel cache files. When this limit is exceeded, any subsequent pixels cached to disk are closed and reopened on demand. This behavior permits a large number of images to be accessed simultaneously on disk, but with a speed penalty due to repeated open/close calls.
- area
- maximum area in bytes of any one image that can reside in the pixel cache memory. If this limit is exceeded, the image is automagically cached to disk.
- memory
- maximum amount of memory in bytes to allocate for the pixel cache from the anonymous mapped memory or the heap.
- map
- maximum amount of memory map in bytes to allocate for the pixel cache.
- disk
- maximum amount of disk space in bytes permitted for use by the pixel cache. If this limit is exceeded, the pixel cache is not created and a fatal exception is thrown.
To determine the current setting of these limits, use this command:
$magick> identify -list resourceFile Area Memory Map Disk Thread Time
------------------------------------------------------------------------
768 11.551GiB 8.6631GiB 23.102GiB 16EiB 8 unlimited
You can set these limits either with environment variables, the -limit command line option, or the SetMagickResourceLimit() MagickCore API method. As an example, our online web interface to ImageMagick, ImageMagick Studio, has an area limit of 64 mebibytes, a memory limit of 128 mebibytes and a map limit of 256 mebibytes and a disk limit of 1 gibibytes. Since we process multiple simultaneous sessions, we don't want any one session consuming all the available memory. Instead large images are cached to disk. If the image is too large and exceeds the pixel cache disk limit, the program exits. In addition, we place a 60 second time limit to prevent any run-away processing tasks.
Note, the cache limits are global, meaning if you create several images, the combined resource requirements are compared to the limit to determine the pixel cache storage disposition.
Cache Views
GetVirtualPixels(), GetAuthenticPixels(), QueueAuthenticPixels(), and SyncAuthenticPixels() from the MagickCore API can only deal with one pixel cache area per image at a time. Suppose you want to access the first and last scanline from the same image at the same time? The solution is to use a cache view. A cache view permits you to access as many areas simultaneously in the pixel cache as you require. The cache view methods behave like the previous methods except you must first open a view and close it when you are finished with it. Here is a snippet of MagickCore code that permits us to access two areas of an image simultaneously:
CacheView
*view_1,
*view_2;
view_1=OpenCacheView(source);
view_2=OpenCacheView(source);
for (y=0; y < (long) source->rows; y++)
{
u=GetCacheViewVirtualPixels(view_1,0,y,source->columns,1,exception);
v=GetCacheViewVirtualPixels(view_2,0,source->rows-y-1,source->columns,1,exception);
if ((u == (const PixelPacket *) NULL) || (v == (const PixelPacket *) NULL))
break;
for (x=0; x < (long) source->columns; x++)
{
/* do something with u & v here */
}
}
view_1=CloseCacheView(view_1);
view_2=CloseCacheView(view_2);
if (y < (long) source->rows)
{ /* an exception was thrown */ }
Magick Persistent Cache Format
Recall that each image format is decoded by ImageMagick and the pixels are deposited in the pixel cache. If you write an image, the pixels are read from the pixel cache and encoded as required by the format you are writing (e.g. GIF, PNG, etc.). The Magick Persistent Cache (MPC) format is designed to eliminate the overhead of decoding and encoding pixels to and from an image format. MPC writes two files. One, with the extension .mpc, retains all the properties associated with the image or image sequence (e.g. width, height, colorspace, etc.) and the second, with the extension .cache, is the pixel cache in the native format. When reading an MPC image file, ImageMagick reads the image properties and memory maps the pixel cache on disk eliminating the need for decoding the image pixels. The tradeoff is in disk space. MPC is generally larger in file size than most other image formats.
Best Practices
Although you can request any pixel from the pixel cache, any block of pixels, any scanline, multiple scanlines, any row, or multiple rows with the GetVirtualPixels(), GetAuthenticPixels(), QueueAuthenticPixels, GetCacheViewVirtualPixels(), GetCacheViewAuthenticPixels(), and QueueCacheViewAuthenticPixels() methods, ImageMagick is optimized to return a few pixels or a few pixels rows at time. There are additional optimizations if you request a single scanline or a few scanlines at a time. These methods also permit random access to the pixel cache, however, ImageMagick is optimized for sequential access.
If you update pixels returned from GetAuthenticPixels() or GetCacheViewAuthenticPixels(), don't forget to call SyncAuthenticPixels() or SyncCacheViewAuthenticPixels() respectively to ensure your changes are synchronized with the pixel cache.
Use QueueAuthenticPixels() or QueueCacheViewAuthenticPixels() if you are setting an initial pixel value. The GetAuthenticPixels() or GetCacheViewAuthenticPixels() method reads pixels from the cache and if you are setting an initial pixel value, this read is unnecessary. Don't forget to call SyncAuthenticPixels() or SyncCacheViewAuthenticPixels() respectively to push your updates to the pixel cache.
GetVirtualPixels(), GetAuthenticPixels(), QueueAuthenticPixels(), and SyncAuthenticPixels() are slightly more efficient than their cache view counter-parts. However, cache views are required if you need access to more than one region of the image simultaneously or if more than one thread of execution is accessing the image.
You can request pixels outside the bounds of the image with GetVirtualPixels() or GetCacheViewVirtualPixels(), however, it is more efficient to request pixels within the confines of the image region.
Although you can force the pixel cache to disk using appropriate resource limits, disk access can be upwards of 1000 times slower than memory access. For fast, efficient, access to the pixel cache, try to keep the pixel cache in heap memory or anonymous mapped memory.
The ImageMagick Q16 version of ImageMagick permits you to read and write 16 bit images without scaling but the pixel cache consumes twice as much resources as the Q8 version. If your system has constrained memory or disk resources, consider the Q8 version of ImageMagick. In addition, the Q8 version typically executes faster than the Q16 version.
A great majority of image formats and algorithms restrict themselves to a fixed range of pixel values from 0 to some maximum value, for example, the Q16 version of ImageMagick permit intensities from 0 to 65535. High dynamic-range imaging (HDRI), however, permits a far greater dynamic range of exposures (i.e. a large difference between light and dark areas) than standard digital imaging techniques. HDRI accurately represents the wide range of intensity levels found in real scenes ranging from the brightest direct sunlight to the deepest darkest shadows. Enable HDRI at ImageMagick build time to deal with high dynamic-range images, but be mindful that each pixel component is a 32-bit floating point value. In addition pixel values are not clamped so some algorithms may perform differently than the non-HDRI version.
If you are dealing with large images, make sure the pixel cache is written to a disk area with plenty of free space. Under Unix, this is typically /tmp and for Windows, c:/temp. You can tell ImageMagick to write the pixel cache to an alternate location with the MAGICK_TEMPORARY_PATH environment variable. For example,
$magick> export MAGICK_TEMPORARY_PATH=/data/magick
If you plan on processing the same image many times, consider the MPC format. Reading a MPC image has near-zero overhead because its in the native pixel cache format eliminating the need for decoding the image pixels. Here is an example:
$magick> convert image.tif image.mpc $magick> convert image.mpc -crop 100x100+0+0 +repage 1.png $magick> convert image.mpc -crop 100x100+100+0 +repage 2.png $magick> convert image.mpc -crop 100x100+200+0 +repage 3.png
MPC is ideal for web sites. It reduces the overhead of reading and writing an image. We use it exclusively at our online image studio.
Streaming Pixels
ImageMagick provides for streaming pixels as they are read from or written to an image. This has several advantages over the pixel cache. The time and resources consumed by the pixel cache scale with the area of an image, whereas the pixel stream resources scale with the width of an image. The disadvantage is the pixels must be consumed as they are streamed so there is no persistence.
Use ReadStream() or WriteStream() with an appropriate callback method in your MagickCore program to consume the pixels as they are streaming. Here's an abbreviated example of using ReadStream:
static size_t StreamHandler(const Image *image,const void *pixels,
const size_t columns)
{
/* process pixels here */
return(columns);
}
...
/* invoke the pixel stream here */
image=ReadStream(image_info,&StreamHandler,exception);
We also provide a lightweight tool, stream, to stream one or more pixel components of the image or portion of the image to your choice of storage formats. It writes the pixel components as they are read from the input image a row at a time making stream desirable when working with large images or when you require raw pixel components.
Image Properties and Profiles
Images have metadata associated with them in the form of properties (e.g. width, height, description, etc.) and profiles (e.g. EXIF, IPTC, color management). ImageMagick provides convenient methods to get, set, or update image properties and get, set, update, or apply profiles. Some of the more popular image properties are associated with the Image structure in the MagickCore API. For example:
(void) printf("image width: %lu, height: %lu\n",image->columns,image->rows);
For a great majority of image properties, such as an image comment or description, we use the GetImageProperty() and SetImageProperty() methods. Here we set a property and fetch it right back:
const char
*comment;
(void) SetImageProperty(image,"comment","This space for rent");
comment=GetImageProperty(image,"comment");
if (comment == (const char *) NULL)
(void) printf("Image comment: %s\n",comment);
Image profiles are handled with GetImageProfile(), SetImageProfile(), and ProfileImage() methods. Here we set a profile and fetch it right back:
StringInfo
*profile;
profile=AcquireStringInfo(length);
SetStringInfoDatum(profile,my_exif_profile);
(void) SetImageProfile(image,"EXIF",profile);
DestroyStringInfo(profile);
profile=GetImageProfile(image,"EXIF");
if (profile != (StringInfo *) NULL)
(void) PrintStringInfo(stdout,"EXIF",profile);
Threads of Execution
Many of ImageMagick's internal algorithms are threaded to take advantage of speed-ups offered by the dual and quad-core processor technologies. However, you are welcome to use ImageMagick algorithms in your threads of execution with the exception of the MagickCore's GetVirtualPixels(), GetAuthenticPixels(), QueueAuthenticPixels(), or SyncAuthenticPixels() pixel cache methods. These methods are intended for one thread of execution only. To access the pixel cache with more than one thread of execution, use a cache view. We do this for the CompositeImage() method, for example. Suppose we want to composite a single image over a different image in each thread of execution. If we use GetVirtualPixels(), the results are unpredictable because multiple threads would likely be asking for different areas of the pixel cache simultaneously. Instead we use GetCacheViewVirtualPixels() which creates a unique view for each thread of execution ensuring our program behaves properly regardless of how many threads are invoked. The other program interfaces, such as the MagickWand API, are completely thread safe so there are no special precautions for threads of execution.
Here is an example of how ImageMagick can take advantage of threads of execution with the OpenMP programming paradigm:
If you call the ImageMagick API from your OpenMP-enabled application and you intend to dynamically increase the number of threads available in subsequent parallel regions, be sure to perform the increase before you call the API otherwise ImageMagick may fault.
Custom Image Coders
An image coder (i.e. encoder / decoder) is responsible for registering, optionally classifying, optionally reading, optionally writing, and unregistering one image format (e.g. PNG, GIF, JPEG, etc.). Registering an image coder alerts ImageMagick a particular format is available to read or write. While unregistering tells ImageMagick the format is no longer available. The classifying method looks at the first few bytes of an image and determines if the image is in the expected format. The reader sets the image size, colorspace, and other properties and loads the pixel cache with the pixels. The reader returns a single image or an image sequence (if the format supports multiple images per file), or if an error occurs, an exception and a null image. The writer does the reverse. It takes the image properties and unloads the pixel cache and writes them as required by the image format.
Here is a listing of a sample custom coder. It reads and writes images in the MGK image format which is simply an ID followed by the image width and height followed by the RGB pixel values.
To invoke the custom coder from the command line, use these commands:
$magick> convert logo: logo.mgk $magick> display logo.mgk
We provide the Magick Coder Kit to help you get started writing your own custom coder.
Custom Image Filters
ImageMagick provides a convenient mechanism for adding your own custom image processing algorithms. We call these image filters and they are invoked from the command line with the -process option or from the MagickCore API method ExecuteModuleProcess().
Here is a listing of a sample custom image filter. It computes a few statistics such as the pixel brightness and saturation mean and standard-deviation.
To invoke the custom filter from the command line, use this command:
$magick> convert logo: -process analyze -verbose info:Image: logo:
Format: LOGO (ImageMagick Logo)
Class: PseudoClass
Geometry: 640x480
...
filter:brightness:kurtosis: 8.98864
filter:brightness:mean: 238.096
filter:brightness:skewness: -3.04519
filter:brightness:standard-deviation: 46.3286
filter:saturation:kurtosis: 5.9137
filter:saturation:mean: 23.4635
filter:saturation:skewness: 2.71874
filter:saturation:standard-deviation: 64.7734
We provide the Magick Filter Kit to help you get started writing your own custom image filter.