streamers haven

Best OBS Encoders Ranked – X264 vs NVENC vs AVC

Encoders are the driving force behind content creation involving audio and video in the modern world. There are many encoding protocols out there, but only four matter when it comes to OBS Studio. Which encoder you end up using will depend on what you are doing, as the best obs encoder for you will vary depending on specific situations involving component utilization.

I say four because support for H265 is currently fairly limited, so practically speaking, there are only four to choose from when factoring in compatibility.

What is a Encoder in OBS?

An encoder, as it relates to OBS Studio, is simply a way to record and package data into a video or audio computer-readable format, like (.wav) or (.mp4). How the encoding is done differs based on what encoder you are using.

What Kind of Encoders Are There?

There are two types of encoders:

  • Software
  • Hardware

You can think of a software encoder like an emulator – it fakes the process using clever programming. As a result, it is not quite as fast as hardware encoders, as it needs to be processed by a separate piece of unspecialized hardware (your CPU). With that said, what it lacks in speed, it makes up for with increased compatibility.

On the other hand, hardware encoding is accomplished using a purpose-built chip that does not need to be processed by the CPU before sending it on its way. This is because each step in the encoding process is hardcoded into a circuit, so the only limiting factors in this process are how well the encoder is laid out and how good each processing circuit is.

How to Select Your Encoder in OBS Studio

OBS Encoder Select
I have an AMD RX480

For those of you new to this, here is a short guide on how to change your encoder in OBS Studio – Below is a quick recap.

  1. Click Settings
    • Or File -> Settings
  2. Select Output
  3. Set Output Mode to Advanced
  4. Under Encoder, click the dropdown
  5. Select your Encoder
  6. Click Apply

To set it for recording, repeat the steps, with one additional step after step 3

  • Select the Recording Tab

Check out my post regarding the best OBS settings to use for recording.

All Encoders Are Equal With Enough Bitrate

When it comes to recording content with ample bitrate, the differences between each encoder fall off sharply. In fact, I’d wager you’d have a really hard time, if not find it impossible to tell the difference between encoders that have sufficient bitrate associated with the resolution and FPS of the encoded content, assuming you don’t overload them or fail to give it enough system resources to do its job.

The differences between them are minute and relate mostly to their efficiency at lower bitrate values, commonly enforced on live streaming platforms like Twitch.

Important Note – Avoid 100% Utilization of CPU and GPU

Best OBS Encoder issues when 100 percent utilization occurs. Heavily corrupted frames in gta 5.
100% Utilization can result in This on stream. Yikes.

The best OBS encoder for you can change based on system resource utilization. Specifically, if the component that is encoding happens to hit 100% utilization, you will experience massive frame drops, lag, frame latency, and a host of other very obvious issues. (See above image)

This happens because both the game (or running program) and the encoder are demanding resources simultaneously, and the poor components just kind of throw in the towel and shoves performance randomly at both. The result is that neither process gets the processing time it needs to properly execute, resulting in very bizarre behavior.

1: The Best OBS Encoder: (New) Nvenc – GPU Encoding

The (New) Nvenc encoder is easily the best OBS encoder available, assuming you have a modern NVIDIA graphics card with the improved ASIC chip. With zero performance impact while encoding and high-quality frames even at low bitrates, it is a strong contender as a streaming encoder.

It wasn’t designed to encode video game content, but it does a pretty good job at it.

Pros:

  • Zero performance impact in game
    • * as long as GPU does not hit 100% utilization
  • Dedicated Encoding chip directly on the card
  • Very few issues with frame stutters
  • Visual artifacts are the least common among these encoding options

Cons:

  • Available only on Select Nvidia GPUS, starting with the GTX 1650 Super, 1660 Ti, and all RTX cards
  • Not Magic?

Reasons Why You Might Not Use (New) Nvenc

The biggest reason why you might not use the (new) Nvenc encoder is that you simply do not have a card that has the chip. This encoder requires that you use one of a select few graphics cards by Nvidia (GTX 1660 TI or better, and any RTX card), and good luck with the silicon shortage going on and price gouging. I had to shell out over $800 to get my 3060 TI. Not a fun experience, let me tell you.

Another reason you might not use this encoder would be because you’re pegging your GPU at 100% utilization. You can help fight this by frame limiting the card, but some of you might not want to lose your precious frame crunching power. In which case, the alternative then is to use the X264 encoder instead. Or, you know, a dedicated streaming PC or rendering rig.

The final reason to not use this encoder is if you are having trouble getting a good final result out of it. As a troubleshooting step, I recommend trying the older NVENC, and if the problem persists, X264. If the problem goes away, try updating your video card drivers.

2: Nvenc – Nvidia GPU Specific Encoder

The Nvenc Encoder is actually an ASIC chip soldered directly onto the PCB of an Nvidia Graphics card. It is completely separate from the GPU itself, with the sole exception of PCI-E bandwidth utilization. As a result, this encoder has literally zero performance impact when encoding a video feed.

Of course, there is one exception to that as well – utilization, but not for the reasons you might expect. If you peg your video card to 100% utilization, things tend to get complicated. It isn’t the encoder itself that runs into issues – It’s OBS, which requires some GPU processing power for the preview window and a few other processes. This results in the encoder beginning to act up in strange ways.

In short, avoid 100% utilization when encoding.

Pros:

  • Zero performance impact in-game
    • * as long as GPU does not hit 100% utilization
  • Dedicated encoding chip directly on the card that is separate from the GPU itself
  • Very few issues with frame stutters
  • Visual artifacts are less common than other encoders

Cons:

  • Only available on NVIDIA GPUs
  • Image quality suffers with lower bitrates (sub 4,000 bitrate)

Reasons Why You Might Not Use Nvenc

As for the biggest reason why you might not use the Nvenc encoder is that your GPU is of AMD or Intel make. This encoder requires that you use an Nvidia graphics card. I’m not 100% about the first series to have the ASIC embedded into it, but suffice it to say that so long as you have a card that is somewhat modern, it has the encoding chip in it. (Modern being within the last three generations.)

Another reason you might not use this encoder would be that you’re pegging your GPU at 100% utilization. This is the only scenario where things become problematic for the encoder. You can help fight this by frame-limiting the card, but some of you might not want to lose your precious frame-crunching power. In this case, the alternative then is to use the X264 encoder instead.

3: X264 – CPU Encoding

X264 encoding is easily the most accessible out of every encoding option in OBS. This is because it is done at a software level, and almost all modern processors are capable of using it. However, because it runs as a software layer, it is far less efficient at processing the frame data. So it uses a lot of CPU computing resources to achieve the desired effect.

This can be especially pronounced when you are streaming a high CPU-demanding game, such as Anno 2077 or other simulation-based titles.

Pros:

  • Available on most modern computers
  • Fairly efficient at low bitrates (in terms of quality)
  • Very few issues with frame stutters
  • Visual artifacts are consistently low

Cons:

  • Reserves some of your CPU overhead (Can be bad for CPU intensive game streaming)
  • Noticable performance fps loss in game while streaming/recording
  • Higher heat load on your CPU

Reasons Why You Might Not Use X264

The biggest reason you might not use the X264 encoder is that it creates a pretty substantial demand on the CPU, resulting in performance loss while encoding. But, of course, the amount of performance lost is dependant on exactly what CPU you have, with lower-end chips taking the biggest hits.

If you happen to have a HEDT class CPU, like one of these AMD 32-core Threadrippers, then you can pretty safely get away with encoding using X264, as the performance impact on one of those monsters is negligible in most cases. Assuming you don’t try to slow the encoder down to, like, very slow or placebo. (Medium or above is recommended)

4: AVC/H264 (AMD Advanced Media Framework)

I really wanted AMD’s encoder to be good for streaming, but the honest truth is that it isn’t. The only reason you should use this encoder is if your processor is too strained to handle the additional load of encoding. Remember, the goal is to keep utilization for both your CPU and GPU under 100% since massive lag and frame drops occur when this happens.

For recording, there is absolutely nothing wrong with this encoder. It performs just as well as all of the others with ample bitrate.

Pros:

  • An alternative to X264
  • Does not use CPU resources to encode

Cons:

  • Experiences the most frame stutters while encoding among these four encoders
  • Poor encoding performance at low bitrates
  • Only Available on AMD video cards
  • Uses some of your Graphics card’s processing power, impacting performance when streaming/recording

Reasons Why You Might Not Use AVC/H264 (AMD Advanced Media Framework)

The biggest reason why you might not use the AVC encoder is that it is pretty bad at encoding video at low bitrates. This makes it a poor choice for live streaming encoders. However, with that said, as I stated earlier in the article, once it has enough bitrate to do what it needs to do, it will perform just as well as the rest of them when it comes to encoding quality.

Regarding Multi-GPU Setups

If you are using a powerful AMD GPU and happen to have an old Nvidia card laying around, you actually can plug it into your pc, and use its encoding chip. However, I can’t recommend that you offload the encoding this way because it increases the demand on your power supply and can create system instability from mixed GPU drivers.

It isn’t worth the extra heat, noise, and power draw. I would sooner recommend that you build a second, dedicated streaming pc to accomplish the task of encoding. Still, nothing is stopping you from trying, I suppose. I’m not responsible for any damage to your PC if you attempt this and things go bad. Just make sure your power supply is higher than 850 watts before you try something like that.

New Encoder on the Horizon – AV1?

As of OBS Studio 27.2, OBS now has beta support for AV1, an open-source CPU-based software encoder. This is pretty big news because the AV1 encoder is far more efficient at encoding frame data than X264. What this means, in practice, is that it will be better at encoding frame data within the limited bitrate constraints imposed by many live streaming platforms. The result – better quality live streaming video footage.

As for where AV1 stands vs NVENC (New), that has yet to be determined. However, for all you AMD or Intel GPU users out there, this might be a game-changer for you.

Here Are Some Popular Choices for Streamer Gear

Hey, thanks for reading the article! So I’ve compiled this small resource for you guys in case you may be on the lookout for some handy or helpful things to add to your streaming setup. Some of you may be new to streaming and may not know about this stuff, so I wanted to bring this stuff to your attention.

There are a large number of cool products designed to make the lives of streamers and content creators easier or to improve the quality of their setup. Before I do list them though, I strongly recommend that you do your research and check reviews from multiple sources, even beyond those I’ve included here. It is never bad to get a second, third, or even fourth opinion before you make an investment.

Microphones: One of the most popular microphones for live streaming is the Elgato Wave 3 or Wave 1. This microphone is great for streamers because it gives you a ton of control over your audio chain, mimicking some of the features of the venerable GoXLR virtually without all the wires and complexity.

Here are some reviews that you can reference so you can decide whether or not you’d like to get one for yourself:

EposVox Wave 3 Review / Harris Heller(Alpha Gaming) Wave 3 Review / Podcastage Wave 3 Review

Audio Interfaces: For those of you who’d like to not be limited to a single microphone option, then you’re in luck because Elgato now makes the Wave XLR Audio Interface. This device allows you to use any XLR microphone, including the ever-popular, but gain hungry SM7B without a cloud lifter, and retain the features of the Wave microphones mentioned above.

Here are some reviews of this audio interface:

EposVox Wave XLR Review / Podcastage Wave XLR Review / Harris Heller Wave XLR Review

Green Screens: A green screen is a common tool used by content creators to give them unparalleled control over their backgrounds for content. Many opt to use a green screen to remove their background entirely and overlay themselves onto the gameplay itself. As for What green screen I recommend, you’ll have to read my article about green screens, because it explains it better than what I can fit here.

Lights: Lighting is super important if you care about the quality of your camera feed from your webcam or any camera for that matter. For one, those of you who rely on your monitor for your main source of light will have inconsistent lighting that changes based on what your screen is displaying. The best part is that almost any light will do, as any light is better than no light.

With that said, there are better lights that are designed for production purposes that have better color accuracy, are brighter, and have more control. You can check out some of them in my top 5 lights article. Also, having a dim light in your background on a camera scene will look better.

Leave a Comment