Hello world!
this post is not a tutorial of how to use MediaCodecto encode/decode, instead, it is a checklist of things to look for when the quality of the video is degraded drastically, after encoding/decoding without any clear explanation.
The basics
Before we get started let’s talk about what Encoding and Decoding are for those of us who don’t know or need a quick refresher.
What is Encoding?
Encoding is the process of converting a given video (multiple images) file into binary,
Encoding also runs the provided video file through compression algorithms to try and keep as much of the video quality as possible while making the file itself smaller. With that being said, the more a video is compressed, the more the quality is reduced. all of this is done using codecs.
The current standard algorithm for encoding video is H.264 or MPEG-4 Part 10. However, there is a new less supported standard called High-Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2.
HEVC offers from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate.
https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding
What is Decoding?
This is basically the opposite of an encoder. It “decodes” encoded videos using codecs. It recreates a video from a binary format – It takes a compressed video file and outputs a playable container (.MOV, .FLV, .MP4, .OGG, .WMV, WebM) that can be played by video players.
Ok, but what exactly is a Codec?
Codec stands for Coder/Decoder. A video codec is a hardware device or some software that performs compression and decompression of video files.
Codecs determine video size, speed, quality, and more.
The Checklist
Ok so now that we got the basics out of the way, let’s get to the video quality checklist on Android (order her does not matter).
1. Check the bitrate
A video bitrate is the number of data in bits that are processed/compressed per second. This usually determines the size and quality of the video. The higher the bitrate, the better the quality but this also increases the file size.
The reason bitrate affects file size is because the file size is determined by (kilobits per second) x duration. 1 byte per second corresponds to 8 bit/s.
The bitrate is set when the MediaFormat for the encoder is created, which looks something like this:
MediaFormat.createVideoFormat(mimeType, width, height).apply {
setInteger(MediaFormat.KEY_BIT_RATE, bitRate)
// .. some other configs
}
2. Check the frame rate* and the I-frames (or Intraframe) interval
Framerate is how fast the images on a video are displayed. For example, have you or have you ever seen someone draw pictures in the bottom right corner of a notebook, and flip through the pages, creating some sort of animated picture? Frame rate would be how fast those pages are flipped.
The I-frame interval configures the number of partial frames (P-Frames) that occur between full frames (I-Frames) in the video stream. For example, in a scene where a door opens and a person walks through, only the movements of the door and the person are stored by the video encoder. The stationary background that occurs in the previous partial frames is not encoded, because no changes occurred in that part of the scene. The stationary background is only encoded in the full frames. Partial frames improve video compression rates by reducing the size of the video. As the I-frame interval increases, the number of partial frames increases between full frames. Higher values are only recommended on networks with high reliability.
https://support.pelco.com/s/article/Definition-of-the-I-Frame-Interval-1538586573328
However, when configuring the encoder, the frame rate does not really do what you would expect. Normally this configuration is set similar to the bitrate.
MediaFormat.createVideoFormat(mimeType, width, height).apply {
// On LOLLIPOP, media format must not contain a KEY_FRAME_RATE.
if (Build.VERSION.SDK_INT != Build.VERSION_CODES.LOLLIPOP) {
setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
}
setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, iFrameRateInterval)
}
On
https://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html#isFormatSupported(android.media.MediaFormat)Build.VERSION_CODES.LOLLIPOP,formatmust not contain a MediaFormat#KEY_FRAME_RATE. Useformat.setString(MediaFormat.KEY_FRAME_RATE, null)to clear any existing frame rate setting in the format.
The combination of frame-rate and i-frame-interval determines how often I-frames (also called “sync frames”) appear in the encoded output. The MediaCodec encoder does not drop frames. If you want to reduce the frame rate, you have to do it manually by sending fewer frames to it. For more info on this see this StackOverflow post.
3. Check if the encoder being used is hardware accelerated
For better performance and quality, you should make sure that you are using hardware-accelerated codecs.
How to check if the codec is hardware accelerated?
After getting the codec, we can check if it is hardware accelerated in a few ways. If the device is not on Android Q and above we will sadly need to rely on the name of the codec, but on devices that are on Android Q and above we have a handy function for this. Hardware codecs are provided by device manufacturers.
private fun isCodecHardwareAccelerated(codec: MediaCodec): Boolean {
return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
codec.codecInfo.isHardwareAccelerated
} else {
if (codec.name.startsWith("OMX.google.")) {
// This is a software codec
false
} else {
// it is a hardware codec
true
}
}
}
4. Check the profile/level
Now the last thing i would like to draw your attention to are profiles and profile levels. I highly recommend taking a look at the Wikipedia page that explains what this is.
TL;DR
The standard defines several sets of capabilities, which are referred to as profiles, targeting specific classes of applications. These are declared using a profile code (profile_idc) and sometimes a set of additional constraints applied in the encoder. The profile code and indicated constraints allow a decoder to recognize the requirements for decoding that specific bitstream. (And in many system environments, only one or two profiles are allowed to be used, so decoders in those environments do not need to be concerned with recognizing the less commonly used profiles.) By far the most commonly used profile is the High Profile.
Profile levels;:
As the term is used in the standard, a “level” is a specified set of constraints that indicate a degree of required decoder performance for a profile. For example, a level of support within a profile specifies the maximum picture resolution, frame rate, and bit rate that a decoder may use. A decoder that conforms to a given level must be able to decode all bitstreams encoded for that level and all lower levels.
So for example, all devices must support SD (Standard definition) profiles. What this means is that all devices MUST support encoding/decoding at at-least 320 X 240 px with a frame rate of 20 fps and a bitrate of 384 Kbps. For more info see the table below for H.264.
Based on Androids source compatibility documentation all devices that support H.264 MUST support Baseline Profile Level 3 and must support SD encoding profiles. For more on these profile standards see here.
One thing you should know is that, by default, if the profile and profile level are not specified, from my testing, most of the time the Baseline profile seems to be the one selected, which is actually not that great in terms of quality.
Extras
Getting all available Encoders and Decoders
To get all codecs that are “suitable for regular (buffer-to-buffer) decoding or encoding”just do the following:
val mediaCodecList = MediaCodecList(MediaCodecList.REGULAR_CODECS)
NOTE: These are the codecs that are returned prior to API 21, using the now deprecated static methods.”
To get all codecs, even ones that are not suitable for regular (buffer-to-buffer) decoding or encoding. These include codecs, for example, that only work with special input or output surfaces, such as secure-only or tunneled-only codecs.
Simply do:
val mediaCodecList = MediaCodecList(MediaCodecList.ALL_CODECS)
I highly recommend to just using the MediaCodecList.REGULAR_CODECS since it is guaranteed that these will work in almost every scenario.
To check if the codec is an encoder or decoder you can simply do:
MediaCodecList(MediaCodecList.REGULAR_CODECS).codecInfos.forEach {
if (it.isEncoder) {
// encoder ;)
} else {
// decoder
}
}
Get Supported Dimensions
Many times we would like to encode/decode in a certain dimension, but that could cause issues with different devices if it is not supported.
There are ways to check if the dimension we want is supported.
For example, if we wanted to get all supported widths for a specific height we can do:
val codecCapabilities = encoder.codecInfo.getCapabilitiesForType(mime) codecCapabilities.videoCapabilities.getSupportedWidthsFor(height)
I highly recommend taking a look HERE for more info on what can be checked.
Conclusion
The media encoding/decoding world is not that straight forward, and it takes a combination of multiple different configurations to get a desired output, i hope that this checklist provides a decent starting point of where to look when in doubt and gives you a bit more understanding.
Until next time, stay frosty!