IMY format

From Netherworld Research
Revision as of 01:23, 8 September 2016 by Krisan (Talk | contribs) (Created page with "The '''IMY format''' is used to describe compressed data. IMY is a lossless compressed file or a chunk of a file. Sometimes entire archives are compressed by splitting them i...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

The IMY format is used to describe compressed data.

IMY is a lossless compressed file or a chunk of a file. Sometimes entire archives are compressed by splitting them into multiple fixed size IMY files. It is based on run-length encoding but references different parts of the file.

Header

struct header {
  u32 magic;
  u16 unk1;
  u16 unk2;
  u16 offset;     // #used to create look up table
  u8  c_type;     // #compression type, currently only type 8 is known
  u8  unk3;
  u16 unk4;
  u16 unk5;
  u8  padding[16];
  u8  info;       // #number of info bytes
};

Decompression

To decompress the file you need to read every info byte, one by one. Every info byte is an instruction which needs to be performed on the stored data and\or the already compressed data.

Cases

There are in total three different cases/instructions based on a read info byte:

  • If (info byte & 0xF0), is bigger or equal to 16.
    • If ( (info byte & 0x80) && (info byte & 0x40) ), is bigger or equal to 192.
      • Case 1: Copy at least one short from the already uncompressed data by checking n shorts.
    • else
      • Case 2: Copy one short from the data by checking n shorts
  • else
    • Case 3: Copy at least one short from the data and move the data pointer to the read amount.

Note that the data pointer, which is initialized with u8 info; + number of info bytes, is only moved in Case 3.

Information Bytes

  • Case 1:
    • The information byte, holds two types of information:
      • [7][6][5][4][3][2][1][0] = shorts_to_copy = (*info byte* & 0x0F) + 1;
    • The index for the look up table, which holds the number of shorts you need:
      • [7][6][5][4][3][2][1][0] = index = (*info byte* & 0x30) >> 4;
    • Look up table, has four entries:
Index Value
0 2
1 offset (from header)
2 offset + 2
3 offset + 2
  • Case 2:
    • The information byte holds what you need to check for the desired short:
      • [7][6][5][4][3][2][1][0] = lookback_bytes = (*info byte*- 16)*2 + 2;

Implementation

Here is one possible solution, written in C++ :

struct imyHeader{
    unsigned int magic_number; // 4

    unsigned short unknown02; // 2
    unsigned short unknown03; // 2 -> 8
    unsigned short streamOffset; //was width // 2
    unsigned char compressionType; // 1
    unsigned char unknown06; // 1
    unsigned short unknown07; //was height // 2
    unsigned short unknown08; //was paletteSize // 2 -> 16
    unsigned int zero0; //padding // 4
    unsigned int zero1; //padding // 4 -> 24
    unsigned int zero2; //padding // 4
    unsigned int zero3; //padding // 4 -> 32
    unsigned short number_of_info_bytes;// 2 -> 34
} __attribute__((packed, aligned(1)));

/*!
 * @param instream, is a wrapper class witch just can read
 * @param outstream, is a wrapper class witch can read and write
 */
bool decompressIMYSimple(PG::STREAM::In* instream, PG::STREAM::InOut* outstream){
    const unsigned int startOffset = outstream->pos();

    //read the header
    imyHeader header;
    instream->read((char*) &header, sizeof(imyHeader));

    if(header.magic_number != 0x00594D49){
        std::cerr << "IMY file magic number is wrong!" << std::endl;
        return FAILURE;
    }

    if( (header.compressionType >> 4) == 1){

        const unsigned int decompressedFileSize = header.streamOffset * header.unknown07; //not sure

        unsigned int infoBytesOffset = instream->pos();
        const unsigned int infoBytesOffsetEnd = infoBytesOffset + header.number_of_info_bytes;
        unsigned int dataOffset = infoBytesOffsetEnd;

        //create look up table
        PG::UTIL::Array<unsigned int, 4> lookUpTable;
        lookUpTable[0] = 2; // go back one short
        lookUpTable[1] = header.streamOffset;
        lookUpTable[2] = lookUpTable[1] + 2;
        lookUpTable[3] = lookUpTable[1] - 2;

        while( (outstream->pos()-startOffset) < decompressedFileSize && infoBytesOffset < infoBytesOffsetEnd){

            // read every info byte
            instream->seek(infoBytesOffset);
            unsigned char infoByte = instream->readChar();
            infoBytesOffset++;

            if( infoByte & 0xF0 ){
                if( (infoByte & 0x80) && (infoByte & 0x40)){
                    //copy shorts from the already uncompressed stream by looking back
                    const int index = (infoByte & 0x30) >> 4; // the value can only be 0-3
                    const int shorts_to_copy = (infoByte & 0x0F) + 1;

                    for (int i = 0; i < shorts_to_copy; i++){
                        //read
                        const unsigned int currentEnd = outstream->pos();
                        outstream->seek(currentEnd - lookUpTable[index]);
                        const short s = outstream->readShort();
                        outstream->seek(currentEnd);
                        outstream->writeShort(s);
                    }

                }else{
                    //copy a short from the compressed stream by looking back to a short
                    const unsigned int lookback_bytes = (infoByte - 16)*2 + 2;
                    instream->seek(dataOffset-lookback_bytes);
                    outstream->writeShort(instream->readShort());
                }
            }else{
                // just copy shorts (2 byte)
                //you always copy at least one short
                const unsigned int copy_bytes = (infoByte+1)*2;
                char c[copy_bytes];
                instream->seek(dataOffset);
                instream->read(&c[0], copy_bytes);
                outstream->write(&c[0], copy_bytes);
                dataOffset += copy_bytes;
            }
        }
    }else{
        std::cerr << "IMY compression type '"<<header.compressionType<<"' is not supported!" << std::endl;
        return FAILURE;
    }

    return SUCCESS;
}