Adding support for new module formats ===================================== This document describes the basics of writing a new module loader and related work that has to be done. We will not discuss in detail how to write the loader, have a look at existing loaders to get an idea how they work in general. General hints ------------- * We strive for quality over quantity. The goal is not to support as many module formats as possible, but to support them as well as possible. * Write defensive code. Guard against out-of-bound values, division by zero and similar stuff. libopenmpt is constantly fuzz-tested to catch any crashes, but of course we want our code to be reliable from the start. * Every format should have its own `MODTYPE` flag, unless it can be reasonably represented as a subset of another format (like Ice Tracker ICE files being a subset of ProTracker MOD). * When reading binary structs from the file, use our data types with defined endianness, which can be found in `common/Endianness.h`: * Big-Endian: (u)int8/16/32/64be, float32be, float64be * Little-Endian: (u)int8/16/32/64le, float32le, float64le Entire structs containing integers with defined endianness can be read in one go if they are tagged with `MPT_BINARY_STRUCT` (see existing loaders for an example). * `CSoundFile::m_nChannels` **MUST NOT** be changed after a pattern has been created, as existing patterns will be interpreted incorrectly. For module formats that support per-pattern channel amounts, the maximum number of channels must be determined beforehand. * Strings can be safely handled using: * `FileReader::ReadString` and friends for reading them directly from a file * `mpt::String::ReadBuf` for reading them from a struct or char array These functions take care of string padding (zero / space padding) and will avoid reading past the end of the string if there is no terminating null character. * Do not use non-const static variables in your loader. Loaders need to be thread-safe for libopenmpt. * `FileReader` instances may be used to treat a portion of a file as its own independent file (through `FileReader::ReadChunk`). This can be useful with "embedded files" such as WAV or Ogg samples. Container formats such as UMX are another good example for this usage. * Samples *either* use middle-C frequency *or* finetune + transpose. For the few weird formats that use both, it may make sense to translate everything into middle-C frequency. * Add the new `MODTYPE` to `CSoundFile::UseFinetuneAndTranspose` if applicable, and see if any effect handlers in `soundlib/Snd_fx.cpp` need to know the new `MODTYPE`. * Do not rely on hard-coded magic numbers. For example, when comparing if an index is valid for a given array, do not hard-code the array size but rather use `std::size` (or `mpt::array_size` in contexts where `std::size` is not usable) or, for ensuring that char arrays are null-terminated, `mpt::String::SetNullTerminator`. Similarly, do not assume any specific quantities for OpenMPT's constants like MAX_SAMPLES, MAX_PATTERN_ROWS, etc. These may change at any time. * Pay attention to off-by-one errors when comparing against MAX_SAMPLES and MAX_INSTRUMENTS, since sample and instrument numbers are 1-based. Prefer using `CSoundFile::CanAddMoreSamples` and `CSoundFile::CanAddMoreInstruments` to check if a given amount of samples or instruments can be added to the file rather than doing the calculations yourself. * Placement of the loader function in the `ModuleFormatLoaders` array in Sndfile.cpp depends on various factors. In general, module formats that have very bad magic numbers (and thus might cause other formats to get mis-interpreted) should be placed at the bottom of the list. Two notable examples are 669 files, where the first two bytes of the file are "if" (which may e.g. cause a song title starting with if ..." in various other formats to be interpreted as a 669 module), and of course Ultimate SoundTracker modules, which have no magic bytes at all. * Avoid use of functions tagged with [[deprecated]]. Probing ------- libopenmpt provides fast probing functions that can be used by library users to quickly check if a file is most likely playable with libopenmpt, even if only a fraction of the file is available (e.g. when streaming from the internet). In order to satisfy these requirements, probing functions should do as little work as possible (e.g. only parse the header of the file), but as much as required to tell with some certainty that the file is really of a certain mod format. However, probing functions should not rely on having access to more than the first `CSoundFile::ProbeRecommendedSize` bytes of the file. * Probing functions **must not** allocate any memory on the heap. * Probing functions **must not** return ProbeFailure or ProbeWantMoreData for any file that would normally be accepted by the loader. In particular, this means that any header checks must not be any more aggressive than they would be in the real loader (hence it is a good idea to not copy-paste this code but rather put it in a separate function), and the minimum additional size passed to `CSoundFile::ProbeAdditionalSize` must not be higher than the biggest size that would cause a hard failure (i.e. returning `false`) in the module loader. * Probing functions **may** return ProbeSuccess for files that would be rejected by a loader after a more thorough inspection. For example, probing functions do not need to verify that all required chunks of an IFF-like file format are actually present, if the header makes it obvious enough that the file is highly likely to be a module. Adding loader to the build systems and various other locations -------------------------------------------------------------- Apart from writing the module loader itself, there are a couple of other places that need to be updated: * Add loader file to `build/android_ndk/Android.mk`. * Add loader file to `build/autotools/Makefile.am`. * Run `build/regenerate_vs_projects.sh` / `build/regenerate_vs_projects.cmd` (depending on your platform) * Add file extension to `installer/filetypes-*.iss`. * Add file extension to `CTrackApp::OpenModulesDialog` in `mptrack/Mptrack.cpp`. * Add format information to `soundlib/Tables.cpp`.