Ogg notes

From Helpful
Jump to: navigation, search
These are primarily notes
It won't be complete in any sense.
It exists to contain fragments of useful information.

Format and parsing details

Ogg

Ogg is a container format. One physical bitstream (file, or streamed bitstream) can contain many logical bytestreams (e.g. video, audio, anything else).


Logical streams are identified with serial numbers, allowing multiple streams to be decoded in parallel.

That means no two streams in the same physical stream should have the same serial number, even if those do not overlap. (This also means that, while the physical Ogg bitstream allows streams to be directly appended, you cannot do so without regard for serial numbers; see notes below)


Physical bitstreams consist of a number of pages, each consisting of a header and a variable number of segments each up to 255 bytes long. The header itself is variable-sized too, in that it contains a variable-sized table recording the length of each segment.

You can read a bitsteam by looking for the capture pattern, decoding the header, and verifying the page CRC; if it verifies, page sync is verified, and the decoder can extract the page data.


Each page contains:

  • 'OggS'
  • header (27 bytes up to and including the lacing entry amount byte)
    • Ogg version (1 byte, can currently only be 0x00) (so you can currently sync on 'OggS\x00')
    • header type (1 byte), a set of bit flags:
      • 0x01 set means continued packet, otherwise fresh
      • 0x02 set means the first packet in a logical stream
      • 0x04 set means the last packet in a logical stream
    • 'absolute granule position' (a 64-bit int). Its meaning differs per content type
    • stream's serial number (a 32-bit int)
    • page sequence number (a 32-bit int)
    • checksum (a 32-bit int)
    • segment table:
      • amount of lacing values that follow (1 byte)
      • that many values (each a byte), often all have value 0xff (255) except the last value.
    • data (as many bytes as the sum of the values in the segment table specifies)

The packet's total size is the header size sum of the segment table's values.


Notes:

  • The checksum should be taken over the packet as a whole, with the header checksum bytes zeroed out. CRC32 execution details are somewhat unusual; see the Ogg specs.
  • Sequence numbers should be unbroken in a stream.
  • Integers are coded LSB
  • The first packet in a logical stream defines its type, e.g. the name and revision of Vorbis, the audio rate and such (as per the Vorbis definition).





Read:

Invalid/corrupt files

The first symptom is usually the fact that a file won't play. When you ask ogginfo about the file, it will report errors.


Note that the xiph ogg decoders (including ogginfo) are somewhat simplistic about reporting errors. They don't try to resync on pages, they just trust the size of peach page, so when that information is incorrect once it will probably fail on the rest of the file (often in a very verbose way).

I wrote a simple script that resyncs on headers, and gives a little more information (...to those informed about the format).


You can get invalid oggs in various ways, including:


Concatentation

The Ogg spec says you can't just concatenate oggs.

Yes, the page-based nature with stream markers makes that trivial and correct at a mechanical level - it's just more pages, and the streams are marked anyway.

However, every ogg has its own stream serial numbers (identifiers), and if two streams have the same serial number it will create ambiguity which will give errors like:

Warning: illegally placed page(s) for logical stream 1
This indicates a corrupt ogg file: Page found for stream after EOS flag.
Warning: sequence number gap in stream 1. Got page 1 when expecting page 881. 
Indicates missing data.

In theory, random serial numbers would make this very unlikely, but things that just assign streams from 0 onwards within a file make this almost certain to happen.


It is still easy to make an ogg appender that simply makes sure this is not a problem.


And actually, two oggs with the same serial number aren't technically very hard to separate after the fact (by looking at other frame details), but it is still a violation of specs, and ogg demuxers are likely to complain.


ID3

Having an program add ID3 tags to your Oggs is, from an Ogg-container-format view, an invalid thing to do, and some players will complain.

You could strip them again, for example with a command-line utility like id3v2.

Other cases

A bug in taglib 1.4 (version from 2005, used by various other programs) caused it to resize and change tag data but apparently not the lacing values in the second page(verify). This leads to the file not playing and ogginfo reporting, among other things:

Warning: Hole in data (4500 bytes) found at approximate offset something bytes. Corrupted ogg.
Warning: Hole in data (9000 bytes) found at approximate offset something bytes. Corrupted ogg.

Followed by:

Warning: sequence number gap in stream 1. Got page 2 when expecting page 1. 
Indicates  missing data.
Warning: discontinuity in stream (1)

And many, many mentions of:

Warning: Could not decode vorbis header packet 1 - invalid vorbis stream (1)

However, in a good number of cases there is no real data corruption, only an invalid header.

I wrote a script to fix this, by rewriting the second page's lacing table based on the actual page content, and recalculating the CRC value.

This fixed many but not all of my affected files. The problem cases were possibly those were the metadata size became larger at the time of the bad change(verify) - or could just be a bug or faulty assumption in my code.