Introduction to H.264: (1) NAL Unit

H.264 / MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC) is a common video compression format developed by ITU-T Video Coding Experts Group (VCEG) and ISO/IEC JTC1 Moving Picture Experts Group (MPEG). Network Abstraction Layer (NAL) and Video Coding Layer (VCL) are the two main concepts in H.264. A H.264 file consists of a number of NAL units (NALU) and each NALU can be classified as VCL or non-VCL. Video data is processed by the codec and packed into NAL units.

NALU in Packet-Transport Protocol V.S. Byte-Stream Format

There are two ways to pack a NAL unit for different systems, Packet-Transport System and Byte-Stream Format. For Packet-Transport systems like RTP, the transport system protocol frames the coded data into different pieces. Hence, the system can easily identify the boundaries of NAL units and we don’t need to add extra start code, which is a waste of resources. This method is usually used in streaming.

However, in other systems, there is no such protocol to separate NAL units. For example, you want to store a H.264 file and decode it on another computer. The decoder has no idea on how to search the boundaries of the NAL units. So, a three-byte or four-byte start code, 0x000001 or 0x00000001, is added at the beginning of each NAL unit. They are called Byte-Stream Format. Hence, the decoder can now identify the boundaries easily. According to ITU-T Recommendation, this format is usually used in Rec. ITU-T H.222.0 | ISO/IEC 13818-1 systems or Rec. ITU-T H.320 systems.

In this series of articles, I will use byte-stream format as an example to introduce H.264. Here is a sample of a H.264 byte-stream. You can see several start codes 0x00000001 here, which means there are several NAL units in this image.

An excerpt of a h.264 stream
Figure 1: A sample of raw H.264 in byte stream format

But there are chances that 0x000001 or 0x00000001 exists in the bit-stream of a NAL unit. So a emulation prevention bytes, 0x03, is presented when there is 0x000000, 0x000001, 0x000002 and 0x000003 to make them become 0x00000300, 0x00000301, 0x00000302 and 0x00000303 respectively. This ensures that no sequence of consecutive byte-aligned bytes in the NAL unit contains a start code prefix. Line 1 and 2 in the image demonstrate two examples. You may also read the next article for details.

First Byte of NALU: NAL Unit Header

In a NALU, the first byte is a header byte indicating the type of data contained in it and other information. The rest of bytes are the payload of a NAL unit. For the header byte, it can be parsed into 3 parts as shown. Let’s take 0x67 on Line 1 of Figure 1 as an example.

HexBinary0x670110 0111
Parse the First Byte of a NAL unit
Figure 2: Parse the First Byte of a NAL unit

(Note: If you don’t understand the descriptor, read this article: Explanation of Descriptors in the ITU-T Publication on H.264 Coding Standard/Recommendation (with example) first)

So, for 0x67, we have:
forbidden_zero_bit = 0,
nal_ref_idc = 3,
nal_unit_type = 7

The 1st bit is forbidden_zero_bit which is used to check whether there is any error occurred during the transmission. 0 means that it is normal while 1 indicates a syntax violation. So, we should always find that forbidden_zero_bit equals 0.

The next 2 bits are nal_ref_idc indicating whether this NAL unit is a reference field / frame / picture. On one hand, if it is a reference field / frame / picture, nal_ref_idc is not equal to 0. According to the Recommendation, non-zero nal_ref_idc specifies that the content of the NAL unit contains a sequence parameter set (SPS), a SPS extension, a subset SPS, a picture parameter set (PPS), a slice of a reference picture, a slice data partition of a reference picture, or a prefix NAL unit preceding a slice of a reference picture. On the other hand, if it is a non-reference field / frame / picture, nal_ref_idc is equal to 0. For any non-zero value, the larger the value, the more the importance of the NAL unit. In this case, it is equal to 3 (0x11) and is a reference field / frame / picture (In fact, it’s a SPS. I’ll tell you more later).

The following 5 bits specify the nal_unit_type. It specifies the type of RBSP data structure contained in the NAL unit as specified in Table 7-1. VCL NAL units are specified as those NAL units having nal_unit_type equal to 1 to 5, inclusive. All remaining NAL units are called non-VCL NAL units.

Additional Reference on NAL Unit Type from the Specification

The following table lists all NAL unit types and their properties. (Normally, we follow Annex A.)

nal_unit_typeContent of NAL unit & RBSP syntax structureCNAL unit type class [Annex A]NAL unit type class [Annex G & H]NAL unit type class [Annex I]
0Unspecifiednon-VCLnon-VCLnon-VCL
1Coded slice of a non-IDR picture
slice_layer_without_partitioning_rbsp( )
2, 3, 4VCLVCLVCL
2Coded slice data partition A
slice_data_partition_a_layer_rbsp( )
2VCLnot applicablenot applicable
3Coded slice data partition B
slice_data_partition_b_layer_rbsp( )
3VCLnot applicablenot applicable
4Coded slice data partition C
slice_data_partition_c_layer_rbsp( )
4VCLnot applicablenot applicable
5Coded slice of an IDR picture
slice_layer_without_partitioning_rbsp( )
2, 3VCLVCLVCL
6Supplemental enhancement information (SEI)
sei_rbsp( )
5non-VCLnon-VCLnon-VCL
7Sequence parameter set
seq_parameter_set_rbsp( )
0non-VCLnon-VCLnon-VCL
8Picture parameter set
pic_parameter_set_rbsp( )
1non-VCLnon-VCLnon-VCL
9Access unit delimiter
access_unit_delimiter_rbsp( )
6non-VCLnon-VCLnon-VCL
10End of sequence
end_of_seq_rbsp( )
7non-VCLnon-VCLnon-VCL
11End of stream
end_of_stream_rbsp( )
8non-VCLnon-VCLnon-VCL
12Filler data
filler_data_rbsp( )
9non-VCLnon-VCLnon-VCL
13Sequence parameter set extension
seq_parameter_set_extension_rbsp( )
10non-VCLnon-VCLnon-VCL
14Prefix NAL unit
prefix_nal_unit_rbsp( )
2non-VCLsuffix dependentsuffix dependent
15Subset sequence parameter set
subset_seq_parameter_set_rbsp( )
0non-VCLnon-VCLnon-VCL
16..18Reservednon-VCLnon-VCLnon-VCL
19Coded slice of an auxiliary coded picture without partitioning
slice_layer_without_partitioning_rbsp( )
2, 3, 4non-VCLnon-VCLnon-VCL
20Coded slice extension
slice_layer_extension_rbsp( )
2, 3, 4non-VCLVCLVCL
21Coded slice extension for depth view components /* specified in Annex I */
slice_layer_extension_rbsp( ) /* specified in Annex I */
2, 3, 4non-VCLnon-VCLVCL
22..23Reservednon-VCLnon-VCLVCL
24..31Unspecifiednon-VCLnon-VCLnon-VCL
Table 7-1 – NAL unit type codes, syntax element categories, and NAL unit type classes

 

nal_unit_type specifies the type of RBSP data structure contained in the NAL unit as specified in Table 7-1.

The column marked “C” lists the categories of the syntax elements that may be present in the NAL unit. In addition, syntax elements with syntax category “All” may be present, as determined by the syntax and semantics of the RBSP data structure. The presence or absence of any syntax elements of a particular listed category is determined from the syntax and semantics of the associated RBSP data structure. nal_unit_type shall not be equal to 3 or 4 unless at least one syntax element is present in the RBSP data structure having a syntax element category value equal to the value of nal_unit_type and not categorized as “All”.

The entry “suffix dependent” for nal_unit_type equal to 14 is specified as follows:
–  If the NAL unit directly following in decoding order is a NAL unit with type 1 or 5, it is a VCL NAL unit.
–  Otherwise (the NAL unit directly following in decoding order is a NAL unit with type not equal to 1 or 5), it is a non-VCL NAL unit.
Decoders shall ignore (remove from the bit-stream and discard) the NAL unit with type 14 and the NAL unit directly following (in decoding order) the NAL unit with type 14.

— from 7.4.1 NAL unit semantics on p. 62

 

For packing into byte-stream format:

byte_stream_nal_unit( NumBytesInNALunit ) {               // Descriptor 
	while(next_bits(24) != 0x000001 &&
		next_bits(32) != 0x00000001)
			leading_zero_8bits /* equal to 0x00 */        // f(8)

	if(next_bits(24) != 0x000001)
		zero_byte /* equal to 0x00 */                     // f(8)
		
	start_code_prefix_one_3bytes /* equal to 0x000001 */  // f(24)
	nal_unit( NumBytesInNALunit )   
	while(more_data_in_byte_stream() &&
		next_bits(24) != 0x000001 &&
		next_bits(32) != 0x00000001)
			trailing_zero_8bits /* equal to 0x00 */       // f(8)
}

Relationship between nal_ref_idc and NAL Unit Type

NAL unit typePossible nal_ref_idc valueRemarks (7.4.1.2.5 ordering rules)
0, 24..31 (unspecified)Not constrained (0–3)May be present in the access unit but shall not precede the first VCL NAL unit of the primary coded picture.
1 (coded slice of a non-IDR picture)0 or non-zero. If 0 for one NAL unit of types 1–4 of a picture, it shall be 0 for all NAL units of types 1–4 of that picture.If arbitrary slice order (ASO) is allowed, slices may be in any order. Otherwise slice groups shall not be interleaved, and slices within a slice group are in increasing order of the first macroblock address. With separate_colour_plane_flag equal to 1, this applies per colour_plane_id, and the order between different colour_plane_id values is not constrained.
2 (slice data partition A)0 or non-zero (same rule as type 1)Ordered like type 1 for ASO and slice groups. A partition A with a given slice_id shall precede any partition B or C with the same slice_id. Without ASO, partition A shall immediately precede the partition B with the same slice_id, or partition C if B is absent. With separate_colour_plane_flag equal to 1, the same applies for the same slice_id and colour_plane_id.
3 (slice data partition B)0 or non-zero (same rule as type 1)Shall follow partition A with the same slice_id. When present, B shall precede any partition C with the same slice_id, and without ASO it shall immediately precede it.
4 (slice data partition C)0 or non-zero (same rule as type 1)Shall follow partition A (and B, if present) with the same slice_id.
5 (coded slice of an IDR picture)Shall not be 0 (non-zero only)If ASO is allowed, slices may be in any order. Otherwise slice groups shall not be interleaved, and slices within a slice group are in increasing order of the first macroblock address. With separate_colour_plane_flag equal to 1, this applies per colour_plane_id. Data partitioning cannot be used for IDR pictures, and an IDR picture shall not contain NAL unit types 1–4.
12 (filler data)Shall be 0May be present in the access unit but shall not precede the first VCL NAL unit of the primary coded picture.
20..23 (slice extension types, reserved types)Not constrained by 7.4.1 (see Annexes G, H and I)Shall not precede the first VCL NAL unit of the primary coded picture within the access unit.
Order of VCL NAL units and association to coded pictures (7.4.1.2.5), with nal_ref_idc constraints from 7.4.1
Nal Unit TypePossible nal_ref_idc valueIf arbitrary slice order is allowed as specified in Annex A, coded slice of an IDR picture NAL units may have any order relative to each other. Sequence parameter setnon-zero8 Picture parameter setnon-zero13 Sequence parameter set extensionnon-zero15 Subset sequence parameter setnon-zero6, 9, 10, 11 or 120

21 thoughts on “Introduction to H.264: (1) NAL Unit”

  1. If you’re going to change to 6E, you have to update this:

    So, for 0x6E, we have:
    forbidden_zero_bit = 0,
    nal_ref_idc = 3,
    nal_unit_type = 7

    Thanks!

    1. Hi, sorry for the mistake again. I change it back to 0x67. Please let me know whether there is any other mistake. Hope you aren’t confused.

  2. how to generate a NAL byte stream for decoding?
    I am working on h.265 video decoder to implement it with verilog HDL. The input to the cabac decoder is byte stream so how i can get the byte stream? please help me.
    –

    1. Sorry for late reply!
      You may convert with some software like ffmpeg.
      Some media-supported chips also provide byte stream directly.

  3. In hexstream format, is there parameter which define RGB of the video frame? Is there represented after the entry_point_offset_minus1 parameter?

    Thank you

  4. ” a three-byte or four-byte start code, 0x000001 or 0x00000001″
    what are the difference between 0x000001 (3Bytes) and 0x00000001(4Bytes)?

    I heard that 0x00000001(4Bytes)meaning:
    a slice contain a full frame。
    0x000001 (3Bytes) meaning:
    a full frame is consist of some slices。

    I don’t find different points of them in h264 specification。

  5. You blog are awesome are more informative

    But I am confused when i analysis source code of node media server parsing rtmp video data payload
    frame_type = (payload[0]>>4)&0x0f;
    codec_id = payload[0]&0x0f;

    Here not using NALU concept why

    I please reply

    1. Hi, thanks for your appreciation but I have switched my focus to other areas in recent years. So.. I am afraid I cannot help.

  6. Hi all,
    is there a way to check if a given VCL nalu (type 5 for example, Coded slice of an IDR picture) references a full image rather than just a slice?
    Thanks

  7. Your image links are broken? Can you fix the broken links? This article is super interesting but hard to understand without the images. Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *