The purpose of this chapter is to provide you with an overview of the ORFEO Toolbox system. We recommend that you read this chapter to gain an appreciation for the breadth and area of application of OTB. In this chapter, we will make reference either to OTB features or ITK features without distinction. Bear in mind that OTB uses ITK as its core element, so all the fundamental elements of OTB come from ITK. OTB extends the functionalities of ITK for the remote sensing image processing comunity. We benefit from the Open Source development approach chosen for ITK, which allows us to provide an impressive set of functionalities with much lesser effort than it would have been the case in a closed source universe!
The ORFEO Toolbox consists of several subsystems. A brief description of these subsystems follows. Later sections in this chapter—and in some cases additional chapters—cover these concepts in more detail. (Note: in the previous chapter, another module—OTB-Documents is briefly described.)
This section describes some of the core concepts and implementation features found in ITK and therefore also in OTB.
Generic programming is a method of organizing libraries consisting of generic—or reusable—software components [96]. The idea is to make software that is capable of “plugging together” in an efficient, adaptable manner. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms such as sorting. Generic programming is implemented in C++ with the template programming mechanism and the use of the STL Standard Template Library [7].
C++ templating is a programming technique allowing users to write software in terms of one or more unknown types T. To create executable code, the user of the software must specify all types T (known as template instantiation) and successfully process the code with the compiler. The T may be a native type such as float or int, or T may be a user-defined type (e.g., class). At compile-time, the compiler makes sure that the templated types are compatible with the instantiated code and that the types are supported by the necessary methods and operators.
ITK uses the techniques of generic programming in its implementation. The advantage of this approach is that an almost unlimited variety of data types are supported simply by defining the appropriate template types. For example, in OTB it is possible to create images consisting of almost any type of pixel. In addition, the type resolution is performed at compile-time, so the compiler can optimize the code to deliver maximal performance. The disadvantage of generic programming is that many compilers still do not support these advanced concepts and cannot compile OTB. And even if they do, they may produce completely undecipherable error messages due to even the simplest syntax errors. If you are not familiar with templated code and generic programming, we recommend the two books cited above.
In ITK and OTB classes are defined by a maximum of two files: a header .h file and an implementation file—.cxx if a non-templated class, and a .txx if a templated class. The header files contain class declarations and formatted comments that are used by the Doxygen documentation system to automatically produce HTML manual pages.
In addition to class headers, there are a few other important header files.
Most classes in OTB are instantiated through an object factory mechanism. That is, rather than using the standard C++ class constructor and destructor, instances of an OTB class are created with the static class New() method. In fact, the constructor and destructor are protected: so it is generally not possible to construct an OTB instance on the heap. (Note: this behavior pertains to classes that are derived from itk::LightObject . In some cases the need for speed or reduced memory footprint dictates that a class not be derived from LightObject and in this case instances may be created on the heap. An example of such a class is itk::EventObject .)
The object factory enables users to control run-time instantiation of classes by registering one or more factories with itk::ObjectFactoryBase . These registered factories support the method CreateInstance(classname) which takes as input the name of a class to create. The factory can choose to create the class based on a number of factors including the computer system configuration and environment variables. For example, in a particular application an OTB user may wish to deploy their own class implemented using specialized image processing hardware (i.e., to realize a performance gain). By using the object factory mechanism, it is possible at run-time to replace the creation of a particular OTB filter with such a custom class. (Of course, the class must provide the exact same API as the one it is replacing.) To do this, the user compiles her class (using the same compiler, build options, etc.) and inserts the object code into a shared library or DLL. The library is then placed in a directory referred to by the OTB_AUTOLOAD_PATH environment variable. On instantiation, the object factory will locate the library, determine that it can create a class of a particular name with the factory, and use the factory to create the instance. (Note: if the CreateInstance() method cannot find a factory that can create the named class, then the instantiation of the class falls back to the usual constructor.)
In practice object factories are used mainly (and generally transparently) by the OTB input/output (IO) classes. For most users the greatest impact is on the use of the New() method to create a class. Generally the New() method is declared and implemented via the macro itkNewMacro() found in Utilities/ITK/Common/itkMacro.h.
By their nature object-oriented systems represent and operate on data through a variety of object types, or classes. When a particular class is instantiated to produce an instance of that class, memory allocation occurs so that the instance can store data attribute values and method pointers (i.e., the vtable). This object may then be referenced by other classes or data structures during normal operation of the program. Typically during program execution all references to the instance may disappear at which point the instance must be deleted to recover memory resources. Knowing when to delete an instance, however, is difficult. Deleting the instance too soon results in program crashes; deleting it too late and memory leaks (or excessive memory consumption) will occur. This process of allocating and releasing memory is known as memory management.
In ITK, memory management is implemented through reference counting. This compares to another popular approach—garbage collection—used by many systems including Java. In reference counting, a count of the number of references to each instance is kept. When the reference goes to zero, the object destroys itself. In garbage collection, a background process sweeps the system identifying instances no longer referenced in the system and deletes them. The problem with garbage collection is that the actual point in time at which memory is deleted is variable. This is unacceptable when an object size may be gigantic (think of a large 3D volume gigabytes in size). Reference counting deletes memory immediately (once all references to an object disappear).
Reference counting is implemented through a Register()/Delete() member function interface. All instances of an OTB object have a Register() method invoked on them by any other object that references an them. The Register() method increments the instances’ reference count. When the reference to the instance disappears, a Delete() method is invoked on the instance that decrements the reference count—this is equivalent to an UnRegister() method. When the reference count returns to zero, the instance is destroyed.
This protocol is greatly simplified by using a helper class called a itk::SmartPointer . The smart pointer acts like a regular pointer (e.g. supports operators -> and ⋆) but automagically performs a Register() when referring to an instance, and an UnRegister() when it no longer points to the instance. Unlike most other instances in OTB, SmartPointers can be allocated on the program stack, and are automatically deleted when the scope that the SmartPointer was created is closed. As a result, you should rarely if ever call Register() or Delete() in OTB. For example:
In this example, reference counted objects are created (with the New() method) with a reference count of one. Assignment to the SmartPointer interp does not change the reference count. At the end of scope, interp is destroyed, the reference count of the actual interpolator object (referred to by interp) is decremented, and if it reaches zero, then the interpolator is also destroyed.
Note that in ITK SmartPointers are always used to refer to instances of classes derived from itk::LightObject . Method invocations and function calls often return “real” pointers to instances, but they are immediately assigned to a SmartPointer. Raw pointers are used for non-LightObject classes when the need for speed and/or memory demands a smaller, faster class.
In general, OTB uses exception handling to manage errors during program execution. Exception handling is a standard part of the C++ language and generally takes the form as illustrated below:
where a particular class may throw an exceptions as demonstrated below (this code snippet is taken from itk::ByteSwapper :
Note that itk::ByteSwapperError is a subclass of itk::ExceptionObject . (In fact in OTB all exceptions should be derived from itk::ExceptionObject.) In this example a special constructor and C++ preprocessor variables __FILE__ and __LINE__ are used to instantiate the exception object and provide additional information to the user. You can choose to catch a particular exception and hence a specific OTB error, or you can trap any OTB exception by catching ExceptionObject.
Event handling in OTB is implemented using the Subject/Observer design pattern [48] (sometimes referred to as the Command/Observer design pattern). In this approach, objects indicate that they are watching for a particular event—invoked by a particular instance–by registering with the instance that they are watching. For example, filters in OTB periodically invoke the itk::ProgressEvent . Objects that have registered their interest in this event are notified when the event occurs. The notification occurs via an invocation of a command (i.e., function callback, method invocation, etc.) that is specified during the registration process. (Note that events in OTB are subclasses of EventObject; look in itkEventObject.h to determine which events are available.)
To recap via example: various objects in OTB will invoke specific events as they execute (from ProcessObject):
To watch for such an event, registration is required that associates a command (e.g., callback function) with the event: Object::AddObserver() method:
When the event occurs, all registered observers are notified via invocation of the associated Command::Execute() method. Note that several subclasses of Command are available supporting const and non-const member functions as well as C-style functions. (Look in Common/Command.h to find pre-defined subclasses of Command. If nothing suitable is found, derivation is another possibility.)
Multithreading is handled in OTB through ITK’s high-level design abstraction. This approach provides portable multithreading and hides the complexity of differing thread implementations on the many systems supported by OTB. For example, the class itk::MultiThreader provides support for multithreaded execution using sproc() on an SGI, or pthread_create on any platform supporting POSIX threads.
Multithreading is typically employed by an algorithm during its execution phase. MultiThreader can be used to execute a single method on multiple threads, or to specify a method per thread. For example, in the class itk::ImageSource (a superclass for most image processing filters) the GenerateData() method uses the following methods:
In this example each thread invokes the same method. The multithreaded filter takes care to divide the image into different regions that do not overlap for write operations.
The general philosophy in ITK regarding thread safety is that accessing different instances of a class (and its methods) is a thread-safe operation. Invoking methods on the same instance in different threads is to be avoided.
OTB; as ITK, uses the VNL numerics library to provide resources for numerical programming combining the ease of use of packages like Mathematica and Matlab with the speed of C and the elegance of C++. It provides a C++ interface to the high-quality Fortran routines made available in the public domain by numerical analysis researchers. ITK extends the functionality of VNL by including interface classes between VNL and ITK proper.
The VNL numerics library includes classes for
Most VNL routines are implemented as wrappers around the high-quality Fortran routines that have been developed by the numerical analysis community over the last forty years and placed in the public domain. The central repository for these programs is the ”netlib” server http://www.netlib.org/. The National Institute of Standards and Technology (NIST) provides an excellent search interface to this repository in its Guide to Available Mathematical Software (GAMS) at http://gams.nist.gov, both as a decision tree and a text search.
ITK also provides additional numerics functionality. A suite of optimizers, that use VNL under the hood and integrate with the registration framework are available. A large collection of statistics functions—not available from VNL—are also provided in the Insight/Numerics/Statistics directory. In addition, a complete finite element (FEM) package is available, primarily to support the deformable registration in ITK.
There are two principal types of data represented in OTB: images and meshes. This functionality is implemented in the classes Image and Mesh, both of which are subclasses of itk::DataObject . In OTB, data objects are classes that are meant to be passed around the system and may participate in data flow pipelines (see Section 3.5 on page 61 for more information).
otb::Image represents an n-dimensional, regular sampling of data. The sampling direction is parallel to each of the coordinate axes, and the origin of the sampling, inter-pixel spacing, and the number of samples in each direction (i.e., image dimension) can be specified. The sample, or pixel, type in OTB is arbitrary—a template parameter TPixel specifies the type upon template instantiation. (The dimensionality of the image must also be specified when the image class is instantiated.) The key is that the pixel type must support certain operations (for example, addition or difference) if the code is to compile in all cases (for example, to be processed by a particular filter that uses these operations). In practice the OTB user will use a C++ simple type (e.g., int, float) or a pre-defined pixel type and will rarely create a new type of pixel class.
One of the important ITK concepts regarding images is that rectangular, continuous pieces of the image are known as regions. Regions are used to specify which part of an image to process, for example in multithreading, or which part to hold in memory. In ITK there are three common types of regions:
The otb::Image class extends the functionalities of the itk::Image in order to take into account particular remote sensing features as geographical projections, etc.
The Mesh class represents an n-dimensional, unstructured grid. The topology of the mesh is represented by a set of cells defined by a type and connectivity list; the connectivity list in turn refers to points. The geometry of the mesh is defined by the n-dimensional points in combination with associated cell interpolation functions. Mesh is designed as an adaptive representational structure that changes depending on the operations performed on it. At a minimum, points and cells are required in order to represent a mesh; but it is possible to add additional topological information. For example, links from the points to the cells that use each point can be added; this provides implicit neighborhood information assuming the implied topology is the desired one. It is also possible to specify boundary cells explicitly, to indicate different connectivity from the implied neighborhood relationships, or to store information on the boundaries of cells.
The mesh is defined in terms of three template parameters: 1) a pixel type associated with the points, cells, and cell boundaries; 2) the dimension of the points (which in turn limits the maximum dimension of the cells); and 3) a “mesh traits” template parameter that specifies the types of the containers and identifiers used to access the points, cells, and/or boundaries. By using the mesh traits carefully, it is possible to create meshes better suited for editing, or those better suited for “read-only” operations, allowing a trade-off between representation flexibility, memory, and speed.
Mesh is a subclass of itk::PointSet . The PointSet class can be used to represent point clouds or randomly distributed landmarks, etc. The PointSet class has no associated topology.
While data objects (e.g., images and meshes) are used to represent data, process objects are classes that operate on data objects and may produce new data objects. Process objects are classed as sources, filter objects, or mappers. Sources (such as readers) produce data, filter objects take in data and process it to produce new data, and mappers accept data for output either to a file or some other system. Sometimes the term filter is used broadly to refer to all three types.
The data processing pipeline ties together data objects (e.g., images and meshes) and process objects. The pipeline supports an automatic updating mechanism that causes a filter to execute if and only if its input or its internal state changes. Further, the data pipeline supports streaming, the ability to automatically break data into smaller pieces, process the pieces one by one, and reassemble the processed data into a final result.
Typically data objects and process objects are connected together using the SetInput() and GetOutput() methods as follows:
In this example the source object itk::RandomImageSource is connected to the itk::ShrinkImageFilter , and the shrink filter is connected to the mapper otb::ImageFileWriter . When the Update() method is invoked on the writer, the data processing pipeline causes each of these filters in order, culminating in writing the final data to a file on disk.
The ITK spatial object framework supports the philosophy that the task of image segmentation and registration is actually the task of object processing. The image is but one medium for representing objects of interest, and much processing and data analysis can and should occur at the object level and not based on the medium used to represent the object.
ITK spatial objects provide a common interface for accessing the physical location and geometric properties of and the relationship between objects in a scene that is independent of the form used to represent those objects. That is, the internal representation maintained by a spatial object may be a list of points internal to an object, the surface mesh of the object, a continuous or parametric representation of the object’s internal points or surfaces, and so forth.
The capabilities provided by the spatial objects framework supports their use in object segmentation, registration, surface/volume rendering, and other display and analysis functions. The spatial object framework extends the concept of a “scene graph” that is common to computer rendering packages so as to support these new functions. With the spatial objects framework you can:
Currently implemented types of spatial objects include: Blob, Ellipse, Group, Image, Line, Surface, and Tube. The itk::Scene object is used to hold a list of spatial objects that may in turn have children. Each spatial object can be assigned a color property. Each spatial object type has its own capabilities. For example, itk::TubeSpatialObject s indicate to what point on their parent tube they connect.
There are a limited number of spatial objects and their methods in ITK, but their number is growing and their potential is huge. Using the nominal spatial object capabilities, methods such as mutual information registration, can be applied to objects regardless of their internal representation. By having a common API, the same method can be used to register a parametric representation of a building with an image or to register two different segmentations of a particular object in object-based change detection.