[MS-RDPRFX]:
Remote Desktop Protocol: RemoteFX Codec Extension
Intellectual Property Rights Notice for Open Specifications Documentation
§ Technical Documentation. Microsoft publishes Open Specifications documentation (“this documentation”) for protocols, file formats, data portability, computer languages, and standards support. Additionally, overview documents cover inter-protocol relationships and interactions.
§ Copyrights. This documentation is covered by Microsoft copyrights. Regardless of any other terms that are contained in the terms of use for the Microsoft website that hosts this documentation, you can make copies of it in order to develop implementations of the technologies that are described in this documentation and can distribute portions of it in your implementations that use these technologies or in your documentation as necessary to properly document the implementation. You can also distribute in your implementation, with or without modification, any schemas, IDLs, or code samples that are included in the documentation. This permission also applies to any documents that are referenced in the Open Specifications documentation.
§ No Trade Secrets. Microsoft does not claim any trade secret rights in this documentation.
§ Patents. Microsoft has patents that might cover your implementations of the technologies described in the Open Specifications documentation. Neither this notice nor Microsoft's delivery of this documentation grants any licenses under those patents or any other Microsoft patents. However, a given Open Specifications document might be covered by the Microsoft Open Specifications Promise or the Microsoft Community Promise. If you would prefer a written license, or if the technologies described in this documentation are not covered by the Open Specifications Promise or Community Promise, as applicable, patent licenses are available by contacting .
§ License Programs. To see all of the protocols in scope under a specific license program and the associated patents, visit the Patent Map.
§ Trademarks. The names of companies and products contained in this documentation might be covered by trademarks or similar intellectual property rights. This notice does not grant any licenses under those rights. For a list of Microsoft trademarks, visit www.microsoft.com/trademarks.
§ Fictitious Names. The example companies, organizations, products, domain names, email addresses, logos, people, places, and events that are depicted in this documentation are fictitious. No association with any real company, organization, product, domain name, email address, logo, person, place, or event is intended or should be inferred.
Reservation of Rights. All other rights are reserved, and this notice does not grant any rights other than as specifically described above, whether by implication, estoppel, or otherwise.
Tools. The Open Specifications documentation does not require the use of Microsoft programming tools or programming environments in order for you to develop an implementation. If you have access to Microsoft programming tools and environments, you are free to take advantage of them. Certain Open Specifications documents are intended for use in conjunction with publicly available standards specifications and network programming art and, as such, assume that the reader either is familiar with the aforementioned material or has immediate access to it.
Support. For questions and support, please contact .
Revision Summary
Date / Revision History / Revision Class / Comments /4/23/2010 / 0.1 / Major / First Release.
6/4/2010 / 1.0 / Major / Updated and revised the technical content.
7/16/2010 / 1.0 / None / No changes to the meaning, language, or formatting of the technical content.
8/27/2010 / 2.0 / Major / Updated and revised the technical content.
10/8/2010 / 3.0 / Major / Updated and revised the technical content.
11/19/2010 / 4.0 / Major / Updated and revised the technical content.
1/7/2011 / 5.0 / Major / Updated and revised the technical content.
2/11/2011 / 6.0 / Major / Updated and revised the technical content.
3/25/2011 / 7.0 / Major / Updated and revised the technical content.
5/6/2011 / 8.0 / Major / Updated and revised the technical content.
6/17/2011 / 9.0 / Major / Updated and revised the technical content.
9/23/2011 / 10.0 / Major / Updated and revised the technical content.
12/16/2011 / 11.0 / Major / Updated and revised the technical content.
3/30/2012 / 12.0 / Major / Updated and revised the technical content.
7/12/2012 / 13.0 / Major / Updated and revised the technical content.
10/25/2012 / 14.0 / Major / Updated and revised the technical content.
1/31/2013 / 15.0 / Major / Updated and revised the technical content.
8/8/2013 / 15.0 / None / No changes to the meaning, language, or formatting of the technical content.
11/14/2013 / 15.0 / None / No changes to the meaning, language, or formatting of the technical content.
2/13/2014 / 16.0 / Major / Updated and revised the technical content.
5/15/2014 / 16.0 / None / No changes to the meaning, language, or formatting of the technical content.
6/30/2015 / 17.0 / Major / Significantly changed the technical content.
10/16/2015 / 17.0 / None / No changes to the meaning, language, or formatting of the technical content.
7/14/2016 / 17.0 / None / No changes to the meaning, language, or formatting of the technical content.
6/1/2017 / 18.0 / Major / Significantly changed the technical content.
9/15/2017 / 19.0 / Major / Significantly changed the technical content.
12/1/2017 / 19.0 / None / No changes to the meaning, language, or formatting of the technical content.
Table of Contents
1 Introduction 6
1.1 Glossary 6
1.2 References 6
1.2.1 Normative References 6
1.2.2 Informative References 7
1.3 Protocol Overview (Synopsis) 7
1.3.1 RemoteFX Codec 7
1.3.1.1 Message Flows 8
1.4 Relationship to Other Protocols 10
1.5 Prerequisites/Preconditions 10
1.6 Applicability Statement 11
1.7 Versioning and Capability Negotiation 11
1.8 Vendor-Extensible Fields 11
1.9 Standards Assignments 11
2 Messages 12
2.1 Transport 12
2.2 Message Syntax 12
2.2.1 Capabilities Messages 12
2.2.1.1 TS_RFX_CLNT_CAPS_CONTAINER 12
2.2.1.1.1 TS_RFX_CAPS 13
2.2.1.1.1.1 TS_RFX_CAPSET 13
2.2.1.1.1.1.1 TS_RFX_ICAP 14
2.2.1.2 TS_RFX_SRVR_CAPS_CONTAINER 15
2.2.1.3 TS_FRAME_ACKNOWLEDGE_CAPABILITYSET 15
2.2.2 Encode Messages 16
2.2.2.1 Common Data Types 16
2.2.2.1.1 TS_RFX_BLOCKT 16
2.2.2.1.2 TS_RFX_CODEC_CHANNELT 16
2.2.2.1.3 TS_RFX_CHANNELT 17
2.2.2.1.4 TS_RFX_CODEC_VERSIONT 18
2.2.2.1.5 TS_RFX_CODEC_QUANT 18
2.2.2.1.6 TS_RFX_RECT 18
2.2.2.2 Encode Header Messages 19
2.2.2.2.1 TS_RFX_SYNC 19
2.2.2.2.2 TS_RFX_CODEC_VERSIONS 19
2.2.2.2.3 TS_RFX_CHANNELS 20
2.2.2.2.4 TS_RFX_CONTEXT 20
2.2.2.3 Encode Data Messages 21
2.2.2.3.1 TS_RFX_FRAME_BEGIN 21
2.2.2.3.2 TS_RFX_FRAME_END 22
2.2.2.3.3 TS_RFX_REGION 22
2.2.2.3.4 TS_RFX_TILESET 23
2.2.2.3.4.1 TS_RFX_TILE 25
2.2.3 Control Messages 26
2.2.3.1 TS_FRAME_ACKNOWLEDGE_PDU 26
3 Protocol Details 28
3.1 Common Details 28
3.1.1 Abstract Data Model 28
3.1.1.1 State Machine 28
3.1.2 Timers 29
3.1.3 Initialization 30
3.1.4 Higher-Layer Triggered Events 30
3.1.5 Processing Events and Sequencing Rules 30
3.1.5.1 Processing the TS_RFX_CLNT_CAPS_CONTAINER Message 30
3.1.6 Timer Events 31
3.1.7 Other Local Events 31
3.1.8 RemoteFX Algorithm 31
3.1.8.1 Encoding 31
3.1.8.1.1 Input Tiling 32
3.1.8.1.2 Differencing (Optional) 32
3.1.8.1.3 Color Conversion (RGB to YCbCr) 32
3.1.8.1.4 DWT 32
3.1.8.1.5 Quantization 34
3.1.8.1.6 Linearization 34
3.1.8.1.7 RLGR Entropy Encoding 36
3.1.8.1.7.1 RLGR1 36
3.1.8.1.7.2 RLGR3 36
3.1.8.1.7.3 RLGR1/RLGR3 Pseudocode 37
3.1.8.1.7.3.1 RLGR1/RLGR3 Decode 37
3.1.8.1.7.3.2 RLGR1/RLGR3 Encode 40
3.1.8.2 Decoding 43
3.1.8.2.1 RLGR Entropy Decoding 44
3.1.8.2.2 Sub-Band Reconstruction 44
3.1.8.2.3 Dequantization 44
3.1.8.2.4 Inverse DWT 44
3.1.8.2.5 Color Conversion (YCbCr to RGB) 44
3.1.8.2.6 Reconstructed Frame 45
3.1.8.3 RemoteFX Stream 45
3.1.8.3.1 Encode Message Sequencing 45
4 Protocol Examples 47
4.1 Sample Use Case 47
4.2 Annotated RemoteFX Messages 48
4.2.1 Capabilities Messages 48
4.2.2 Encode Header Messages 49
4.2.3 Encode Data Messages 50
4.2.4 Sample Decode Data Flow Sequence 53
4.2.4.1 Input TS_RFX_TILESET Message 53
4.2.4.2 Entropy Decoded Data 56
4.2.4.2.1 Y Component Data 57
4.2.4.2.2 Cb Component Data 61
4.2.4.2.3 Cr Component Data 64
4.2.4.3 Inverse Quantization/DWT 68
4.2.4.3.1 Level-3 Sub-bands 68
4.2.4.3.1.1 LL3 69
4.2.4.3.1.2 HL3 69
4.2.4.3.1.3 LH3 69
4.2.4.3.1.4 HH3 70
4.2.4.3.1.5 Inverse DWT-X (LL3 - HL3) 70
4.2.4.3.1.6 Inverse DWT-X (LH3 – HH3) 71
4.2.4.3.1.7 Inverse DWT-Y (L2 – H2) 72
4.2.4.3.2 Level-2 Sub-bands 73
4.2.4.3.2.1 LL2 73
4.2.4.3.2.2 HL2 73
4.2.4.3.2.3 LH2 74
4.2.4.3.2.4 HH2 75
4.2.4.3.2.5 Inverse DWT-X (LL2 – HL2) 77
4.2.4.3.2.6 Inverse DWT-X (LH2 - HH2) 79
4.2.4.3.2.7 Inverse DWT-Y (L1 – H1) 81
4.2.4.3.3 Level-1 Sub-bands 85
4.2.4.3.3.1 LL1 85
4.2.4.3.3.2 HL1 85
4.2.4.3.3.3 LH1 89
4.2.4.3.3.4 HH1 93
4.2.4.3.3.5 Inverse DWT-X (LL1 - HL1) 97
4.2.4.3.3.6 Inverse DWT-X (LH1 - HH1) 104
4.2.4.3.3.7 Inverse DWT-Y (L0 – H0) 112
4.2.4.3.4 Reconstructed Y Component 127
4.2.4.3.5 Reconstructed Cb Component 127
4.2.4.3.6 Reconstructed Cr Component 142
4.2.4.4 Inverse Color Conversion 157
4.2.4.5 Decoded Image 165
5 Security 167
5.1 Security Considerations for Implementers 167
5.2 Index of Security Parameters 167
6 Appendix A: Product Behavior 168
7 Change Tracking 169
8 Index 170
1 Introduction
The Remote Desktop Protocol: RemoteFX Codec Extension is an extension to the Remote Desktop Protocol: Basic Connectivity and Graphics Remoting (as specified in [MS-RDPBCGR]). The RemoteFX Codec Extension specifies a lossy image codec that can be used to encode screen images by using efficient and effective compression.
Sections 1.5, 1.8, 1.9, 2, and 3 of this specification are normative. All other sections and examples in this specification are informative.
1.1 Glossary
This document uses the following terms:
blit: Also known as block image transfer. An operation in which a rectangular block of pixels in a source image is copied onto a destination image.
discrete wavelet transform (DWT): A mathematical procedure that can be used to derive a discrete representation of a signal.
entropy coding: A lossless data compression scheme used to generate compression codes for input symbols based on their statistical properties.
inverse discrete wavelet transform (IDWT): A mathematical procedure that can be used to reconstruct a signal without loss of information.
little-endian: Multiple-byte values that are byte-ordered with the least significant byte stored in the memory location with the lowest address.
quantization: A technique used to reduce a range of values to a single representative value.
tile-based transform: A transform technique in which an input image is segmented into a grid of disjoint tiles and the transform is then applied separately to each individual tile.
YCbCr color space: A color space where each color is represented as a triplet (Y,Cb,Cr), where Y stands for the Luma (brightness) component and Cb,Cr stand for the two Chroma (color) components.
MAY, SHOULD, MUST, SHOULD NOT, MUST NOT: These terms (in all caps) are used as defined in [RFC2119]. All statements of optional behavior use either MAY, SHOULD, or SHOULD NOT.
1.2 References
Links to a document in the Microsoft Open Specifications library point to the correct section in the most recently published version of the referenced document. However, because individual documents in the library are not updated at the same time, the section numbers in the documents may not match. You can confirm the correct section numbering by checking the Errata.
1.2.1 Normative References
We conduct frequent surveys of the normative references to assure their continued availability. If you have any issue with finding a normative reference, please contact . We will assist you in finding the relevant information.
[ARLGR] Malvar, H.S., "Adaptive Run-Length / Golomb-Rice Encoding of Quantized Generalized Gaussian Sources with Unknown Statistics", Proceedings of the Data Compression Conference, 2006 (DCC 2006) pp. 23 - 32, March 2006, http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1607237
[MS-RDPBCGR] Microsoft Corporation, "Remote Desktop Protocol: Basic Connectivity and Graphics Remoting".
[MS-RDPEGDI] Microsoft Corporation, "Remote Desktop Protocol: Graphics Device Interface (GDI) Acceleration Extensions".
[MS-RDPNSC] Microsoft Corporation, "Remote Desktop Protocol: NSCodec Extension".
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997, http://www.rfc-editor.org/rfc/rfc2119.txt
[T123] ITU-T, "Network-Specific Data Protocol Stacks for Multimedia Conferencing", Recommendation T.123, May 1999, http://www.itu.int/rec/T-REC-T.123/en
Note There is a charge to download the specification.
[T125] ITU-T, "Multipoint Communication Service Protocol Specification", Recommendation T.125, February 1998, http://www.itu.int/rec/T-REC-T.125-199802-I/en
Note There is a charge to download the specification.
[X224] ITU-T, "Information technology - Open Systems Interconnection - Protocol for Providing the Connection-Mode Transport Service", Recommendation X.224, November 1995, http://www.itu.int/rec/T-REC-X.224-199511-I/en
Note There is a charge to download the specification.
1.2.2 Informative References
None.
1.3 Protocol Overview (Synopsis)
The Remote Desktop Protocol: RemoteFX Codec Extension reduces the bandwidth associated with desktop remoting by efficiently compressing images. This is achieved by using the RemoteFX codec. The following sections provide an overview of this codec.
1.3.1 RemoteFX Codec
One of the core requirements of desktop remoting is the ability to efficiently compress server-side screen images so that they can be transported over a network and displayed on a client screen. Any codec used for this purpose needs to be able to deliver effective compression (to reduce network bandwidth requirements) and operate with low-latency (to enable efficient interactions with remoted content). A typical desktop screen contains textual content (synthetic images) along with video and photographic content (natural images). Given the sensitivity of the human eye to the sharp features present in textual content, any applied compression has to be visually lossless; otherwise the text will appear blurred.
The RemoteFX codec has been designed to achieve efficient compression, satisfying the goals of high quality and low latency while using a modest amount of computing resources. It is a tile-based transform codec. The transform chosen was a discrete wavelet transform (DWT) because it enables superior compression performance when compressing textual bitmap regions at high quality. The entropy coding is performed using the Run-Length Golomb-Rice Coder (RLGR) ([ARLGR] section 3), which yields compression gains at relatively low computing requirements. The core functional blocks of the RemoteFX codec are shown in the following diagram.
Figure 1: Core functional blocks of the RemoteFX codec
1.3.1.1 Message Flows
RemoteFX codec messages are transported in order over a lossless transport such as TCP/IP. The message syntax has been designed with this prerequisite.