Rust on Ivan Molodetskikh’s Webpage

Motion Blur for Half-Life Video Recording with Vulkan

Sat, 19 Aug 2023 11:25:00 +0400

Video of a Half-Life movement map run, recorded with motion blur.

Half-Life is an award-winning first-person shooter from 1998. One of the many things setting it apart is a fluid player movement system. You can gradually build up speed in the air by turning left and right, and if you jump as soon as you hit the ground, you keep all of that speed.

These techniques, called strafing and bunny-hopping, unlocked a whole new dimension to the game, and spawned a big community of mappers and players. There are hundreds of custom maps with challenging obstacles designed to test one’s movement skills, and thousands of players competing to finish them as fast as possible.

The main hubs for these maps and records are Xtreme-Jumps¹ and Cosy-Climbing. They are centered around Counter-Strike 1.6, which has similar movement mechanics to Half-Life, but paced slower with a jump-speed limit and some tweaks to the acceleration.

People generally record their runs to in-game demo files. They are very lightweight, making them easy to store and share. Players can then capture demos to videos. There are YouTube channels uploading videos of the best, or notable runs and edits, like the Xtreme-Jumps channel.

Since this gameplay style is focused on movement, it has been customary to use motion blur for video recordings. When done right, it can make the video look smoother and nicer to watch; here’s such an edit (volume warning) for example. Though, ultimately, motion blur is a subjective preference, and plenty of people will tell you how it is completely unwatchable. Part of this, I suspect, comes from frequent cases of motion blur done wrong. I shudder at the vision of someone dropping a 60 frames-per-second (FPS) video into a 30 FPS Vegas project, and leaving frame blending on its default enabled setting.

How Motion Blur Works #

The simplest way to get motion blur is to capture the video at a higher frame rate, then blend together multiple frames for every output video frame. And when dealing with a closed-source game engine, there actually isn’t much else you can do. Thankfully, Half-Life demos play back smoothly at any frame rate, regardless of the FPS the player had when running², so this approach works perfectly.

Let’s explore how several frames combine into the final frame. The following interactive widget shows the final frame at the top, and a set of sixty consecutive sub-frames, or samples, below. Green outlines show which of the samples are blended together to form the final frame. Even though motion blur is best experienced in motion, this still-frame demo should make it clear how it works under the hood.

Please visit the blog post page to view this interactive element.

There are two sliders corresponding to two of the most important parameters for controlling the motion blur.

The top one controls the video capturing FPS, usually called “samples per second”, or “SPS” for short, in the context of frame blending. A low SPS value will produce clearly visible copies of the same objects, making it easy to tell apart the individual frames in the final composite. A high SPS value on the other hand will make the composite smooth and nice to look at in the final video.

The bottom slider controls the amount of blur. It is called “exposure” because it is analogous to exposure in photography and filming. Higher exposure means more sub-frames are blended together. Compare low exposure giving a mostly-clear still frame and high exposure resulting in a lot of blur as all samples are included.

Real-world exposure lets the camera sensor take in light continuously while the shutter is open, but game video capturing is discrete and can only use the samples that were recorded. This is why low SPS makes individual sub-frames visible—they are the only samples we have, with no in-between contents available.

So, high SPS is extremely important for good motion blur. When the player is moving or spinning fast, you need a lot of samples to blend together a smooth-looking frame without obvious edges. This means recording the video at a very high FPS. For Half-Life movement-oriented maps, you want to go up to at least around 3600 FPS to get a decent result regardless of what’s happening on the screen. If you can only capture the video at 60 real-time FPS, it will take you sixty times longer to capture the demo at 3600 FPS compared to its real-time duration. That is, an hour of recording per every minute of the demo. Basically, you really want your recording to go as fast as your computer allows, without being limited to real-time.

This problem was one of the main drivers for me to explore and make my own Half-Life video recording tools, first hl-capture, then bxt-rs. Frame blending is included in the widely-used HLAE tool, which has been around for many years. Unfortunately, HLAE had extremely slow recording speed back then (it is considerably better nowadays), making it unreasonable to use high FPS for motion blur. This is why so many of the older movement videos have clearly visible frame blending artifacts.

In the previous post about bxt-rs video recording, I wrote about how these kinds of tools can capture video much faster than real-time. Now let’s see how I added motion blur to the process.

Frame Blending in bxt-rs #

Frame blending is really just per-pixel averaging of multiple frames. If you know anything about how GPUs work, you can immediately see that this is the perfect task to do on a GPU: a ton of mutually-independent computations that can be done in parallel. Better still, the input frames are already on the GPU: the game has just rendered them. This means that we can avoid the costly data transfers between the GPU video memory and the main memory, making the whole process practically instantaneous. In fact, the interactive widget above is basically doing the same computation in real-time, as you’re dragging the sliders.

In bxt-rs I copy the frame rendered by the game into a texture shared between OpenGL and Vulkan. The Vulkan part then converts the pixel values from the RGB format to YUV 4:2:0 used in video files, downloads the result into the main memory, and hands it over to FFmpeg for video encoding.

The frame blending step will live in Vulkan, between receiving a new frame from OpenGL and pixel format conversion. We only need to output and encode the final blended frame, so intermediate sub-frames can stay in RGB.

Color Accumulation #

We don’t actually need all sixty or so sub-frames at once in video memory. We can allocate one buffer for the output frame (the sampling buffer), then accumulate the intermediate frames into it one by one as they come in. Averaging pixel values means summing them up and dividing by the total count, which is equivalent to dividing every value by the total count first, and summing them together afterwards. Since we know the total count in advance (it’s SPS multiplied by exposure), we can do this division and addition as the frames come in, and get the correct averaged result in the end.

I chose the 16-bit RGB format for the sampling buffer. This allows up to 257 sub-frames with no precision loss due to quantization³, which is more than enough. By default, bxt-rs records at 7200 FPS, which means up to 120 sub-frames with an output FPS of 60, or up to 240 sub-frames with an output FPS of 30—both below 257.

let create_info = vk::ImageCreateInfo {
    // Use the 16-bit RGB format in UNORM mode.
    format: vk::Format::R16G16B16A16_UNORM,
    usage:
        // For updating during the sampling stage.
        | vk::ImageUsageFlags::STORAGE
        // For clearing.
        | vk::ImageUsageFlags::TRANSFER_DST
        // For reading during YUV conversion.
        | vk::ImageUsageFlags::SAMPLED,
    // ...
};
let image_sample = device.create_image(&create_info, None)?;

UNORM lets me treat pixel values as floating point numbers from 0 to 1 in the shader, without having to round them manually.

The shader itself is pretty simple. Its inputs are the new sub-frame from the game, the sampling buffer and the weight factor. The code loads a pixel from the new sub-frame, multiplies it by the weight, and adds it into the sampling buffer.

#version 450

// Copied from the color conversion shader where it gave the best performance.
#define WORKGROUP_SIZE 4
layout (local_size_x = WORKGROUP_SIZE, local_size_y = WORKGROUP_SIZE, local_size_z = 1) in;

// Frames coming from the game are rgba8.
layout (binding = 0, rgba8) uniform readonly image2D image_frame;
// Our intermediate buffer is rgba16.
layout (binding = 1, rgba16) uniform image2D image_sample;

// Weight is set for every frame at runtime.
layout (push_constant) uniform push_constants {
    float weight;
};

void main() {
    ivec2 size = imageSize(image_frame);
    uint width = uint(size.x), height = uint(size.y);

    uint x = gl_GlobalInvocationID.x, y = gl_GlobalInvocationID.y;
    if (x >= width || y >= height)
        return;

    ivec2 coords = ivec2(x, y);

    vec4 frameColor = imageLoad(image_frame, coords);
    vec4 sampleColor = imageLoad(image_sample, coords);

    // Accumulate the frame into the intermediate buffer.
    vec4 newColor = sampleColor + frameColor * weight;
    imageStore(image_sample, coords, newColor);
}

Using multiplication instead of division is just a common thing to do. A usual justification is that it’s faster this way. I’m not sure how much this is still true, especially for shader code⁴, but anyhow.

I can use image2D rather than sampler2D for the input image, since in this shader I read every pixel independently of the rest. This lets me avoid making the whole Vulkan combined image sampler required for a sampler2D.

The weight can vary between frames, so I pass it as a Vulkan push constant. This is because exposure can cut a sub-frame in half, causing its weight to be reduced. That might seem like an edge case which can be solved by rounding the exposure to always cover full sub-frames. You can indeed do that for capturing demos. However, bxt-rs also supports capturing TAS playback where the game’s FPS cannot be changed, so the frame blending code must work with any timing thrown at it.

Recording Loop #

As you may find in the previous post, the video recording loop without frame blending looks somewhat like this:

Every frame is copied to an intermediate buffer, because we don’t know its duration until the next frame. Then, bxt-rs converts its colors from RGB to YUV 4:2:0 on the GPU, transfers the result to the main memory, and sends it to FFmpeg for encoding.

With frame blending, the steps are a bit different:

After obtaining the frame duration, bxt-rs computes the weight and accumulates the sub-frame into the sampling buffer. For most frames this is where the process ends. The recorder keeps track of accumulated frame time. As soon as we’ve accumulated a full output frame worth of sub-frames, it’s sent through the usual steps of color conversion and FFmpeg encoding. The sampling buffer is then cleared to zeroes, ready to accumulate for the next video frame.

There are a few edge cases to take care of. First, as I mentioned before, the sub-frame can be covered by the exposure time only partially. This is solved simply by reducing its accumulation weight: instead of full sub-frame durationexposure, the weight becomes exposed sub-frame durationexposure.

Then, a sub-frame may actually be split in half by an output frame boundary, and even span several output frames all by itself. For example, this can happen when a TAS uses a long 4 FPS (0.25 ms) frame for a trick, which needs to be recorded as several output frames in a 60 FPS video. Turns out this is also easy to handle: since we store the frame in an intermediate buffer, we can just tell the sampling code to accumulate it multiple times with different weights, while writing output frames in between.

Synchronization #

Since we store the sub-frame in an intermediate buffer, the accumulation step can run in parallel to the rendering and capturing of the next game frame. Here, “in parallel” means both on CPU (with a second thread) and on GPU. We therefore need to synchronize the GPU access to the intermediate buffer, so that the accumulation step won’t try to read from it at the same time as the capturing step is writing to it.

For that we use the Vulkan memory barriers⁵. The intermediate frame processing in the accumulation step starts with a transfer → shader barrier and ends with a shader → transfer barrier.

let intermediate_frame_barrier = vk::ImageMemoryBarrier::builder()
    // Make the memory write performed by a transfer available.
    .src_access_mask(vk::AccessFlags::TRANSFER_WRITE)
    // Make the memory visible for a shader read.
    .dst_access_mask(vk::AccessFlags::SHADER_READ)
    .old_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
    .new_layout(vk::ImageLayout::GENERAL)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .image(self.image_acquired)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });
self.device.cmd_pipeline_barrier(
    self.command_buffer_accumulate,
    vk::PipelineStageFlags::TRANSFER,
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*intermediate_frame_barrier],
);

// Do the processing...

let intermediate_frame_barrier = vk::ImageMemoryBarrier::builder()
    // We were not writing anything, no memory write to make available.
    .src_access_mask(vk::AccessFlags::empty())
    // Make the memory visible to a transfer write.
    .dst_access_mask(vk::AccessFlags::TRANSFER_WRITE)
    .old_layout(vk::ImageLayout::GENERAL)
    .new_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .image(self.image_acquired)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });
self.device.cmd_pipeline_barrier(
    self.command_buffer_accumulate,
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::PipelineStageFlags::TRANSFER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*intermediate_frame_barrier],
);

Note that this command buffer effectively returns the intermediate frame back to the same state it was in. This trick lets me accumulate either once or multiple times in a row without dealing with any sort of synchronization edge cases.

But wait, the intermediate frame is not the only resource used for accumulation. What about the sampling buffer? It has two possible paths: either it can go for color conversion, where it will be read from, or we might accumulate into it again next sub-frame. Turns out this is not a problem: we can set the destination access mask on the closing barrier to both shader reads (if we go for color conversion) and shader writes (if we accumulate more).

// There's no opening barrier, actually,
// as the sampling buffer is already synchronized.

// Do the processing...

let sampling_buffer_barrier = vk::ImageMemoryBarrier::builder()
    // Make the accumulation shader writes available.
    .src_access_mask(vk::AccessFlags::SHADER_WRITE)
    // Make the memory visible to both shader reads and writes.
    .dst_access_mask(vk::AccessFlags::SHADER_READ | vk::AccessFlags::SHADER_WRITE)
    .old_layout(vk::ImageLayout::GENERAL)
    .new_layout(vk::ImageLayout::GENERAL)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .image(self.image_sample)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });
self.device.cmd_pipeline_barrier(
    self.command_buffer_accumulate,
    // This barrier will wait for the compute shader to finish.
    vk::PipelineStageFlags::COMPUTE_SHADER,
    // The next compute shader should wait for this barrier.
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*sampling_buffer_barrier],
);

We use two buffer, and we synchronized both of them. We’re done, right?

That’s what I thought when I wrote this code, and the validation layers seemed to agree. However, recording a video crashed my AMD driver a few seconds into the process.

After trying several things, re-reading the Vulkan synchronization blog-post and quadruple-checking my barriers, I was out of ideas. I was too lazy to make a C reproducer for a Mesa bug report, so I added a fence wait, blocking the CPU until the GPU finishes the accumulation step. This had a performance cost, but seemed to fix the issue.

// XXX: As far as I can tell, waiting for a fence here
// should not be required (and it makes the process quite
// slower). Unfortunately, I'm getting GPU fence timeouts if
// I don't do it. Maybe there's a synchronization bug in the
// code, but I don't see it, and the validation layers are
// silent too.
let create_info = vk::FenceCreateInfo::default();
let fence = self.device.create_fence(&create_info, None)?;

let command_buffers = [self.command_buffer_accumulate];
let submit_info = vk::SubmitInfo::builder().command_buffers(&command_buffers);
self.device
    .queue_submit(self.queue, &[*submit_info], fence)?;

{
    // I use tracing spans with tracing-chrome for profiling.
    let _span = info_span!("wait for fence").entered();

    // Block everything and wait until the GPU finishes the work.
    self.device
        .wait_for_fences(&[fence], true, u64::max_value())?;
}

self.device.destroy_fence(fence, None);

Several months passed, players were recording videos with motion blur just fine, but The Fence kept sitting in my head. This recording code is supposed to be really fast, and I have this useless idle time right in the middle of the hottest loop.

Just look at the performance profile⁶ with no Fence:

The X axis on these graphs represents time, and colored rectangles correspond to particular ranges of code that I’ve annotated, usually functions. So if the rectangle is short, that means the function took little time, and the longer a rectangle is, the longer that function took. There are two distinct rows on the graph separated with a gray line—these are the two threads: the recording thread at the top and the main game thread at the bottom.

This graph above shows four sub-frames during one of the recording sessions. All empty space between the occasional colored area is the game’s own processing. That is, my video recording logic takes almost no time at all—less than 10% of overhead.

Now compare that to The Fence:

Ouch! Almost all bxt-rs time is the wait, and sometimes it even eats into (and delays!) the next frame’s processing. It’s helped only by the fact it happens on another thread. Certainly in other recording scenarios this overhead will be much worse. Hopefully you can see why this Fence had firmly leased a plot in the back of my mind.

Around that time I was testing my relatively slow ThinkPad T495s with a 4K screen. It’s got no discrete GPU and its AMD Ryzen 5 3500U CPU is an ultra-low-power model, which means lower performance. I was occasionally getting random GPU crashes while using the desktop. I found an AMDGPU bug report about it and jumped in to help diagnose the problem.

There seemed to be several underlying issues at play, but for my laptop I found a consistent reproducer with a heavy Blender file. The developers pinned one of the issues down to a bug in the threaded graphics driver implementation. The workaround was to disable the threads with GALLIUM_THREAD=0, which for me fully fixed the Blender reproducer.

This Blender crash visually looked exactly the same as my sampling video recording crash, so naturally I decided to give it a go. I removed The Fence, set GALLIUM_THREAD=0, started recording my test demo with frame blending. It recorded fine. Hooray, mystery solved! It was a Mesa threading bug. I committed the fence removal and documented the workaround in the README.

A week later an NVIDIA user reports corrupted video frames during recording.

…

Well.

Let’s try with The Fence back, I guess.

No corruption.

At this point I was rather lost. For me, the validation layers were completely silent no matter what I tried, both when my GPU worked fine and when it crashed. I decided to give it one last chance, and asked my NVIDIA user friend to record without the fence, and with the validation layers.

2023-07-11T15:31:23.886949Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xaded368, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xaded368[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.887111Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xaded368[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
2023-07-11T15:31:23.888095Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xadf1188, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xadf1188[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.888272Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xadf1188[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
2023-07-11T15:31:23.912888Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xaded368, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xaded368[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.913049Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xaded368[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
2023-07-11T15:31:23.914085Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xadf1188, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xadf1188[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.914266Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xadf1188[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
2023-07-11T15:31:23.920305Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xaded368, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xaded368[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.920462Z  INFO acquire:acquire_image: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xaded368[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
2023-07-11T15:31:23.920610Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0xadf1188, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer() on active VkCommandBuffer 0xadf1188[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
2023-07-11T15:31:23.920780Z  INFO accumulate:accumulate{weight=0.008333333767950535}: bxt_rs::vulkan: ERROR VALIDATION Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0xaed7188, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0xadf1188[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)

…oh. The important part here is: vkBeginCommandBuffer() on active VkCommandBuffer 0xaded368[] before it has completed. Without The Fence, my recording is calling accumulate so fast that the GPU shader from the last call is still running. 🤦 To make matters worse, the same apparently happens with copying the game’s frame to the intermediate buffer. The Fence avoided the issue by waiting for the shader completion right after starting it.

Well! The proper fix was rather straightforward, and in fact pretty similar to The Fence. We still have to wait for the GPU to finish, but we don’t have to wait right after starting the work. Instead, we can get a bit of parallelization back by waiting right before starting the next batch of work. This gets most of the fence-less performance back while working perfectly fine on both AMD and NVIDIA. The Mesa bug was not involved after all.

Here’s the performance graph to confirm it:

Looks almost the same as the fence-less graph above, save for the tiny pink areas where the GPU is waiting for the previous frame’s command buffer.

Actually, further into the recording my GPU starts exhibiting some funny behavior where the waits gradually increase, and then reset back to nothing at an output frame boundary:

My guess here is that the GPU heats up and starts downclocking, so the shaders start taking longer to run. And at an output frame boundary the long main memory transfer gives the GPU some breathing room to cool back down.

And on my NVIDIA user friend’s setup the wait times seem straight up random:

Well, either way, the issue is fixed, and I’m parallelizing as much as possible, as confirmed by the performance measurements.

I’m still not sure why the validation layers didn’t catch the bug on my system. Maybe they weren’t recent enough, so this check hadn’t been added yet? Regardless, it seems a good idea to run the validation layers on different drivers and GPU vendors before declaring the code working fine.

Overhead #

Measuring overall video recording performance in Half-Life is difficult because, when the recording overhead is sufficiently low, the speed is fully dominated by how fast your computer can actually play a particular demo file. It’s especially challenging on AMD GPUs which are known to have worse performance for legacy OpenGL games such as Half-Life, compared to NVIDIA. Like, here on my system I can look into one corner of the map and get 2000 FPS, then look into a different corner and get 200 FPS.

To get a somewhat reasonable measure of performance, I picked a specific demo (my kz_ytt_ancient TAS, same as the video at the very top of this post) and played it with and without capturing on two of my laptops. I used a resolution of 2560×1440 with 60 output FPS, which is a fairly normal recording target for today.

For these measurements I used the Tracy profiler, which is a low-overhead game-oriented profiler. It automatically shows histograms and statistics for frame times and regions, making it very convenient for performance analysis.

First, the results on my fast laptop: Lenovo Legion 7 Gen 7 AMD with a Ryzen 7 6800H CPU with an RX 6700M discrete GPU. Here I used 7200 SPS, the bxt-rs default. During recording, the demo is played at a fixed timestep of 7200 FPS, so to measure the recording overhead, for the non-recording run I also play the demo at 7200 FPS.

Let’s start with normal demo playback with no recording. Please note that all histograms below use logarithmic scale for the X axis.

These are frame time statistics: they show a histogram of how long each individual frame took to process, as well as the mean and the median, and the equivalent FPS values.

As you can see there’s quite a bit of variation in frame times (note the logarithmic scale!), but the median frame time is 1.21 ms, or 828 FPS. This is measured across 211,876 total frames.

Next, let’s run the same demo while recording a video with motion blur.

The distribution looks similar, but there’s a tail on the right. This tail contains the one-in-120 frames which complete the sampling buffer and cause an output frame to be transferred to the main memory and sent to FFmpeg for encoding. However, this happens infrequently enough, so the median frame time increased by just 0.16 ms, or 160 µs, compared to no recording.

Let’s look at the overhead specifically.

These are statistics for just the bxt-rs recording code. They correspond to extra work done on the main thread during recording on top of the normal game rendering.

As you can see, this overhead is mostly sitting at around 113 µs, just like the frame time difference told us. And once again there’s a small tail on the right, corresponding to the output frame main memory transfer and encoding.

Overall this overhead accounts for 27.89 s out of 4:12.9 of recording time, or about 11% for this demo and setup.

Now let’s look at the results from my ThinkPad T495s with its Ryzen 5 3500U CPU and no discrete GPU. I reduced the SPS from 7200 to 1800 here because the rendering was just too slow unfortunately, and I didn’t feel like waiting that long.

First, the statistics without video recording.

You can immediately see the FPS taking a 4× drop compared to the beefier laptop. There’s also a tail on the right: I found that the map contains one specific spot that the integrated GPU really struggles with. In fact, it’s clearly visible as red on the frame times chart that Tracy provides:

Now let’s look at the graphs while recording with motion blur.

Here you can see the median frame time increased from 4.92 ms all the way up to 15.32 ms. Looks like bxt-rs recording has a sizeable overhead on this setup.

Hm, doesn’t look too bad, just 1.36 ms in the median. Now, one thing to remember is that since we’re recording at four times fewer SPS, we’re having a long output frame once every 30 frames rather than once every 120 frames, so they have a larger compounding impact.

But it seems that there’s a more important issue at play. My guess is that the GPU work for accumulating the sampling buffer every frame overlaps with and slows down the GPU work that goes into rendering the game—the integrated GPU isn’t powerful enough to do both at the same time without a slowdown.

Looking at the fence wait statistics, which show how long the second thread is waiting for the last frame’s GPU work to finish, seems to confirm this:

While plenty of waits are very short (the previous frame’s GPU work had already completed by the time the next frame has started, so there’s no wait), there’s also a big chunk in the higher ranges, up to 10 ms. Compare this to the statistics on the fast laptop:

Yeah, the discrete GPU has got no problem at all doing sampling accumulation and game rendering at the same time.

To sum up, for this particular testing scenario I’m seeing about 15% recording overhead for a laptop with a discrete GPU, and about 200% recording overhead for a laptop with a weak integrated GPU. Since with this setup the fast laptop manages around 730 real-time FPS during recording, and the demo plays back at 7200 FPS, the whole process goes about ten times slower than real-time.

Sidenote: I love how snappy everything in Tracy is. All UI zooming and panning and resizing works in real-time with little dropped frames on my 165 Hz screen, despite the seemingly huge amounts of data involved. I guess Tracy developers put their own work to excellent use on Tracy itself.

Conclusion #

I released video recording with motion blur as part of bxt-rs 4.0. It ended up a bit overshadowed by the big new piece of functionality that is the TAS Editor 2, but most of the people who needed video recording have already been using the development snapshots anyway.

There have been a bunch of videos recorded with this motion blur implementation already, and I’m glad to see that it had caught on, with some channels using it even for the most recent uploads. This is mainly thanks to my aforementioned NVIDIA user friend recommending the tool to people in the Counter-Strike circles, while also implementing several features they’ve been asking for.

I personally used the motion blur recording for my most recent TASes at 8K60 resolution just fine with no issues. This is roughly how fast that goes:

Remember that this is recording at 8K resolution, that is 7680×4320 pixels, at 7200 FPS. The short pauses you might notice are the moments when an output frame is completed and sent over to the main memory and to FFmpeg. Furthermore, this is on an AMD GPU which is known to have poorer performance with the legacy OpenGL used by Half-Life. Pretty fast all things considered.

The final output video for that is https://youtu.be/GnUh6b5NxNo.

Of course, there’s still potential for improvement.

For one thing, as I noted in the previous post, it’s possible to do encoding on the GPU using a hardware encoder. This will eliminate the CPU transfer of the full unencoded frame, which is very likely the main contributing factor to those short pauses you see on the video above. For very high resolutions it would also be potentially faster than libx264 (although I guess it would run into the hardware encoder size limits), while allowing to use newer compression standards, like AV1, which are much more efficient at high resolution.

Then there’s the question of SPS. My chosen default of 7200 SPS is probably way too high for most demos, and you can get the same quality video with lower values. This would mean the game has fewer frames to render, which is the biggest bottleneck in the recording process.

However, it’s quite difficult to estimate a good SPS value ahead of time as it would require something like computing the optical flow (pixel movement vectors) for every frame of the demo to find the largest jump. You have to do it ahead of time because Half-Life’s demo interpolation for some reason really dislikes it when you change the FPS mid-playback, and produces very uneven videos. I went with a default SPS this high as a safe option, because the relatively small recording overhead allows me.

One very interesting idea is exposure greater than a single frame. It might not make much sense with the real world film, but in-game you just need to take some samples from the previous frames and accumulate them into the next frame. This would allow, for example, having the same amount of blur on a 120 FPS video as on a 60 FPS video with a full-frame exposure. A full-frame exposure on a 120 FPS video is equivalent to a half-frame exposure on a 60 FPS video, so right now the best you can get is half the blur.

Unfortunately, as far as I can tell, the general multi-frame exposure case requires keeping around all individual sub-frames to be able to pick and accumulate them for the next frame. If exposure is limited to whole frames, then it should be possible to only keep around one partially-accumulated frame for every whole frame of exposure, which is much more reasonable, but still complicated to implement.

Either way, I’m glad to have this code released and this blog post out. It’s been a good Vulkan diving practice and an interesting parallelism and correctness exercise. I hope you enjoyed the read and learned something useful!

The interactive widget above uses WebGL and a frame averaging shader to simulate the sampling process in real-time. You can take a look at the page source code, it shouldn’t be too complicated.

All 60 input frames are packed into one big 3840×3600 atlas which the shader reads from. Apparently, Chrome on Android (but not Firefox on Android) limits texture sizes to 4096×4096 for all devices due to an issue on some specific GPUs, so the atlas had to fit into that limit.

The whole interactive widget thing was inspired by the fantastic, spectacular posts by Bartosz Ciechanowski. Make sure to check them out at https://ciechanow.ski/ if you somehow haven’t stumbled upon them yet.

Who unfortunately seem to have just recently lost access to their domain, https://xtreme-jumps.eu. They’re in process of rebuilding the site from scratch. ↩︎
This is possible thanks to interpolation done by the Half-Life engine during demo playback. When watching a demo, the engine looks at recorded demo frames around the current playback timestamp, and interpolates the player’s position and angles, as well as other entities’ positions and angles. ↩︎
Frames coming from the game are in 8-bit RGB format, which means the lowest color component value is ¹⁄₂₅₅. The 16-bit RGB format expands this to represent values as low as ¹⁄₆₅₅₃₅. When averaging together 257 color component values through accumulation, we divide every incoming value by 257. In the worst case we end up with the resulting value of ¹⁄₂₅₅ × ¹⁄₂₅₇ = ¹⁄₆₅₅₃₅, which is still representable in our intermediate RGB-16 format. Many frames will have pixel values bigger than the one-above-black ¹⁄₂₅₅, which can be divided by even higher numbers while remaining representable. ↩︎
Even if it matters only for CPU code, it is still useful, as bxt-rs implements the same operations with the same multiplication on the CPU for the fallback path. ↩︎
I heavily recommend this blog post explaining how to use Vulkan synchronization and memory barriers. ↩︎
I use tracing and tracing-chrome to generate Chrome-tracing-compatible performance traces, then view them in Perfetto. ↩︎

Fast Half-Life Video Recording with Vulkan

Tue, 27 Jul 2021 16:51:00 +0300

Half-Life is a famous first-person shooter released in 1998. These days Half-Life has a vibrant speedrunning community where players try their best to complete the game as fast as humanly possible. Single-sitting speedruns, when people play through the game from start to finish, are usually streamed and recorded live. But segmented or tool-assisted speedruns—big projects that take weeks or months to complete—use demo files, which capture player actions and can be played back in-game. The demos are recorded to videos post-factum, typically at high resolution and quality. This video recording of demos can be done faster than real-time, which can save multiple real-life hours.

In this post I’ll show how I implemented a tool capable of recording 1920×1080@60 Half-Life footage at up to 3 times faster than real-time on a reasonably powerful PC.¹ The tool uses the Vulkan graphics API to capture the game’s frames and convert their color space before encoding.

I’ll also show an example of how to share an image between OpenGL and Vulkan, as the information on this admittedly uncommon use-case is still rather scarce.

Recording a 1920×1080@60 video of a Half-Life speedrun demo in 3× real-time

Specialized Recording Tools #

One question you might immediately ask is: why is a special tool needed for recording video game footage, what’s the difference from screen recording tools like Open Broadcaster Software? The answer is that while those tools are great for live video recording, when dealing with in-game demos, we can do much better. In-game demos contain gameplay recordings and can be deterministically played back at any speed, which means that:

We can record them into videos much faster than real-time because we control the demo playback speed.
We can set the target video frame-rate and record at that frame-rate exactly, guaranteeing there are no frame drops, even if the computer can’t keep up in real-time.
We’re not wasting any performance while recording: every single frame the game draws goes into the video.
We can add special effects, for example motion blur by recording at a much higher frame-rate and blending the frames together.

Prior Art #

The most well-known Half-Life video recording tool is Half-Life Advanced Effects, or HLAE for short. It has lots of movie-making features, such as custom camera paths or depth map dumping, but its video recording is fairly basic. HLAE gets the frame buffer contents with glReadPixels and passes them to FFmpeg for encoding.

HLAE implemented the FFmpeg encoding only in summer of 2020. Prior to that it would dump every single frame as a TGA file, which was a big recording performance bottleneck and lead to massive disk space consumption (uncompressed frames weigh a lot!). With FFmpeg the performance is much better, but still far from optimal.

Another video recording tool is my previous project, hl-capture. It used OpenCL to capture frames and convert them to a YUV color space, then passed them to FFmpeg for encoding. The performance was very good, but due to lack of OpenCL-OpenGL interoperability implementation in Mesa, this fast path only worked on NVIDIA GPUs.

Implementation #

I implemented the video recording on top of bxt-rs, a base library which hooks into the Half-Life process and provides utilities for intercepting functions and interfacing with the game. I won’t go into too much detail here; the main point is that bxt-rs allows me to add in-game console commands and run my own code when the game calls some function.

Overview #

Video recording comprises several steps. After Half-Life draws a frame with OpenGL, we need to take the frame’s pixel data, feed it to a compute shader which will convert the colors from RGB to YUV, then send the YUV frame to an FFmpeg process for encoding.

To make recording as fast as possible, we want to overlap the maximum amount of work with the game rendering the next frame. To do that, the only operation we will do on Half-Life’s main thread is copying the frame contents into our own GPU buffer. The game will then continue working on the next frame, while we process the recorded contents on a second thread. Luckily, Vulkan was designed with multithreading in mind and provides the necessary guarantees and synchronization primitives.

All operations on the second thread use Vulkan. We will create a Vulkan image for storing the pixel data and share it with OpenGL on the main thread. Once the main thread copies the frame contents into that image, the second thread will run the color conversion compute shader, transfer the result from the GPU memory into RAM and send it off to FFmpeg.

Choosing the Vulkan Crate #

In my past Rust projects I used both raw Vulkan bindings and vulkano. The raw bindings were, expectedly, way too verbose and boilerplate-heavy. Vulkano on the other hand was very nice to work with, however for this project I wanted a lower level of control to better understand the inner workings of the system I was building. Besides, the external memory extension that I was planning to use for interoperating with OpenGL is rather new, and Vulkano hasn’t yet added support for it. Thus, for interfacing with Vulkan, I went with the ash crate.

Ash turned out to be the perfect fit for the project. It is essentially the raw Vulkan API, made as convenient as possible to use from Rust. Ash had only partial support for the external memory extension, but due to the low-level nature of the crate, I could use the underlying Vulkan functions directly for the parts that ash didn’t have wrappers for.

Choosing the GPU #

As a small aside, I don’t recommend tinkering with Vulkan on AMD (at least on Linux) because the drivers, following Vulkan’s raw nature, do only a bare minimum of checks. Practically, this means your session will crash very frequently, leaving you with a pretty picture like this:

After a few days I ended up temporarily switching to my previous GPU, an NVIDIA GTX 970, and finishing the Vulkan part there. NVIDIA’s proprietary driver proved to be very stable and hasn’t crashed once during my development.

Memory sharing between graphics APIs works by allocating a piece of GPU memory in one API and sending a “pointer” to it to the other API. Specifically, we will create an image in Vulkan and export the underlying memory to OpenGL, where we will bind it to an OpenGL texture. We will also need a shared semaphore to synchronize access to the image between the APIs.

The Vulkan side uses the VK_KHR_external_memory and VK_KHR_external_semaphore extensions to export a memory handle and a semaphore respectively, then the OpenGL side imports them using the EXT_external_objects extension. On Linux exporting is done via file descriptors, and on Windows via handles.

This is how we create an image for sharing with OpenGL:

#[cfg(unix)]
let mut external_memory_image_create_info = vk::ExternalMemoryImageCreateInfo::builder()
    .handle_types(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD);
#[cfg(windows)]
let mut external_memory_image_create_info = vk::ExternalMemoryImageCreateInfo::builder()
    .handle_types(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32);

let create_info = vk::ImageCreateInfo::builder()
    // ...
    .usage(
        vk::ImageUsageFlags::TRANSFER_SRC
            | vk::ImageUsageFlags::TRANSFER_DST
            | vk::ImageUsageFlags::SAMPLED
            | vk::ImageUsageFlags::STORAGE
            | vk::ImageUsageFlags::COLOR_ATTACHMENT
            | vk::ImageUsageFlags::INPUT_ATTACHMENT,
    )
    .push_next(&mut external_memory_image_create_info);

let image_frame = unsafe { device.create_image(&create_info, None)? };

The usage flags include all possible usage types as per a note in EXT_external_objects:

The agreed resolution is to use the final option: Require all supported usage flags be specified by the application on the Vulkan side if the image is intended to alias with an OpenGL texture.

Then we allocate memory for the image:

#[cfg(unix)]
let mut export_memory_allocate_info = vk::ExportMemoryAllocateInfo::builder()
    .handle_types(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD);
#[cfg(windows)]
let mut export_memory_allocate_info = vk::ExportMemoryAllocateInfo::builder()
    .handle_types(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32);

let mut memory_dedicated_allocate_info =
    vk::MemoryDedicatedAllocateInfo::builder().image(image_frame);

let create_info = vk::MemoryAllocateInfo::builder()
    // ...
    .push_next(&mut export_memory_allocate_info)
    .push_next(&mut memory_dedicated_allocate_info);

let image_frame_memory = unsafe { device.allocate_memory(&create_info, None)? };

// Bind our image to the allocated memory.
unsafe { device.bind_image_memory(image_frame, image_frame_memory, 0)? };

Allocating memory for external use requires checking the dedicated memory allocation flag. It can either suggest, or require a dedicated allocation, but it cannot require a non-dedicated allocation. Thus for simplicity I always do a dedicated allocation.

Now let’s export a handle to this memory. I mentioned that ash has partial support for external memory: specifically, it supports Linux file descriptor exporting.

let external_memory_fd = ash::extensions::khr::ExternalMemoryFd::new(instance, &device);

let create_info = vk::MemoryGetFdInfoKHR::builder()
    .memory(image_frame_memory)
    .handle_type(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD);
let memory_handle = unsafe { external_memory_fd.get_memory_fd(&create_info)? };

For Windows, we have to use the raw API directly:

let external_memory_win32_fn = vk::KhrExternalMemoryWin32Fn::load(|name| unsafe {
    mem::transmute(instance.get_device_proc_addr(device.handle(), name.as_ptr()))
});

let create_info = vk::MemoryGetWin32HandleInfoKHR::builder()
    .memory(image_frame_memory)
    .handle_type(vk::ExternalMemoryHandleTypeFlags::EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32);

let mut memory_handle = std::ptr::null_mut();
unsafe {
    external_memory_win32_fn.get_memory_win32_handle_khr(
        device.handle(),
        &*create_info,
        &mut memory_handle,
    )
};

Finally, we can import the handle on the OpenGL side:

let mut memory_object = 0;
gl.CreateMemoryObjectsEXT(1, &mut memory_object);

// We're using a dedicated allocation.
gl.MemoryObjectParameterivEXT(memory_object, gl::DEDICATED_MEMORY_OBJECT_EXT, &1);

#[cfg(unix)]
gl.ImportMemoryFdEXT(
    memory_object,
    size,
    gl::HANDLE_TYPE_OPAQUE_FD_EXT,
    memory_handle,
);
#[cfg(windows)]
gl.ImportMemoryWin32HandleEXT(
    memory_object,
    size,
    gl::HANDLE_TYPE_OPAQUE_WIN32_EXT,
    memory_handle,
);

// Create a texture and bind it to the imported memory.
let mut texture = 0;
gl.GenTextures(1, &mut texture);

gl.BindTexture(gl::TEXTURE_2D, texture);
gl.TexStorageMem2DEXT(
    gl::TEXTURE_2D,
    1,
    gl::RGBA8,
    width,
    height,
    memory_object,
    0,
);

Now we have a Vulkan image and an OpenGL texture backed by the same GPU memory. This means that we can write pixels into the OpenGL texture, then read them back from the Vulkan image just fine, as long as we do synchronization correctly.

This is where a semaphore comes into play. The semaphore can be signaled from one API upon completion of a GPU operation (like writing pixels), and waited-for in the other API before starting another GPU operation (like reading pixels back). This way we can guarantee that the reading operation sees the completed result of the writing operation with no data races.

We’ll do a similar exporting-importing dance for the semaphore, created on the Vulkan side:

#[cfg(unix)]
let mut export_semaphore_create_info = vk::ExportSemaphoreCreateInfo::builder().handle_types(
    vk::ExternalSemaphoreHandleTypeFlags::EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_FD,
);
#[cfg(windows)]
let mut export_semaphore_create_info = vk::ExportSemaphoreCreateInfo::builder().handle_types(
    vk::ExternalSemaphoreHandleTypeFlags::EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32,
);

let create_info =
    vk::SemaphoreCreateInfo::builder().push_next(&mut export_semaphore_create_info);
let semaphore = unsafe { device.create_semaphore(&create_info, None)? };

In this case ash doesn’t include a wrapper for either Linux or Windows, so we use the raw API for both:

// Linux
let external_semaphore_fd_fn = vk::KhrExternalSemaphoreFdFn::load(|name| unsafe {
    mem::transmute(instance.get_device_proc_addr(device.handle(), name.as_ptr()))
});

let create_info = vk::SemaphoreGetFdInfoKHR::builder()
    .semaphore(semaphore)
    .handle_type(vk::ExternalSemaphoreHandleTypeFlags::EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_FD);
let mut semaphore_handle = -1;
unsafe {
    external_semaphore_fd_fn.get_semaphore_fd_khr(
        device.handle(),
        &*create_info,
        &mut semaphore_handle,
    )
};

// Windows
let external_semaphore_win32_fn = vk::KhrExternalSemaphoreWin32Fn::load(|name| unsafe {
    mem::transmute(instance.get_device_proc_addr(device.handle(), name.as_ptr()))
});

let create_info = vk::SemaphoreGetWin32HandleInfoKHR::builder()
    .semaphore(semaphore)
    .handle_type(vk::ExternalSemaphoreHandleTypeFlags::EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32);
let mut semaphore_handle = std::ptr::null_mut();
unsafe {
    external_semaphore_win32_fn.get_semaphore_win32_handle_khr(
        device.handle(),
        &*create_info,
        &mut semaphore_handle,
    )
};

Importing from the OpenGL side is simple:

let mut semaphore = 0;
gl.GenSemaphoresEXT(1, &mut semaphore);

#[cfg(unix)]
gl.ImportSemaphoreFdEXT(semaphore, gl::HANDLE_TYPE_OPAQUE_FD_EXT, semaphore_handle);
#[cfg(windows)]
gl.ImportSemaphoreWin32HandleEXT(
    semaphore,
    gl::HANDLE_TYPE_OPAQUE_WIN32_EXT,
    semaphore_handle,
);

Capturing a Frame #

Here’s a simplified diagram of the Half-Life’s game loop operation:

We’ll capture the pixel contents of a frame by running some code just before the game swaps buffers to show it on the screen:

We need to synchronize access to the shared memory between OpenGL and Vulkan. The memory will be written by OpenGL, then read by Vulkan, then written by OpenGL again next frame, then read by Vulkan again, and so on in a loop. On each of these steps the corresponding API will “acquire” the memory by waiting for the semaphore, do the operation, then “release” the memory by signaling the semaphore again.

On the OpenGL side I blit (copy) the current framebuffer (which contains the frame Half-Life is about to show on screen) to the OpenGL texture bound to the shared memory via an auxiliary framebuffer. I am also careful to restore the previously-bound framebuffer so the game continues working properly.

// Acquire shared memory ownership from Vulkan.
gl.WaitSemaphoreEXT(
    semaphore,
    0,
    null(),
    1,
    [texture].as_ptr(),
    // We keep the image in the general layout between using it.
    [gl::LAYOUT_GENERAL_EXT].as_ptr(),
);

// Save previously-bound framebuffer.
let mut previous_framebuffer = 0;
gl.GetIntegerv(gl::DRAW_FRAMEBUFFER_BINDING, &mut previous_framebuffer);

// Bind a destination framebuffer targeting our texture.
gl.BindFramebuffer(gl::DRAW_FRAMEBUFFER, framebuffer);
gl.FramebufferTexture2D(
    gl::DRAW_FRAMEBUFFER,
    gl::COLOR_ATTACHMENT0,
    gl::TEXTURE_2D,
    texture,
    0,
);

// Copy pixel contents from the game's framebuffer to ours.
gl.BlitFramebuffer(
    0,
    0,
    width,
    height,
    0,
    0,
    width,
    height,
    gl::COLOR_BUFFER_BIT,
    gl::NEAREST,
);

// Restore the framebuffer that the game had bound as the destination.
gl.BindFramebuffer(gl::DRAW_FRAMEBUFFER, previous_framebuffer as u32);

// Transfer shared memory ownership to Vulkan.
gl.SignalSemaphoreEXT(
    semaphore,
    0,
    null(),
    1,
    [texture].as_ptr(),
    // Transfer it back to the general layout.
    [gl::LAYOUT_GENERAL_EXT].as_ptr(),
);

// Submit the commands to the GPU.
gl.Flush();

On the Vulkan side, acquisition and release with another graphics API (OpenGL) require barriers:

let begin_info =
    vk::CommandBufferBeginInfo::builder().flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT);
device.begin_command_buffer(command_buffer_sampling, &begin_info)?;

// Acquire the image from OpenGL.
let image_frame_memory_barrier = vk::ImageMemoryBarrier::builder()
    .src_access_mask(vk::AccessFlags::empty())
    .dst_access_mask(vk::AccessFlags::empty())
    // We keep the image in the general layout between using it.
    .old_layout(vk::ImageLayout::GENERAL)
    .new_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL)
    // QUEUE_FAMILY_EXTERNAL indicates a different graphics API.
    .src_queue_family_index(vk::QUEUE_FAMILY_EXTERNAL)
    .dst_queue_family_index(queue_family_index)
    .image(image_frame)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });

device.cmd_pipeline_barrier(
    command_buffer_sampling,
    vk::PipelineStageFlags::TOP_OF_PIPE,
    vk::PipelineStageFlags::TRANSFER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*image_frame_memory_barrier],
);

// (additional commands to blit the image to another intermediate buffer)

// Transfer image_frame back to OpenGL.
let image_frame_memory_barrier = vk::ImageMemoryBarrier::builder()
    .src_access_mask(vk::AccessFlags::empty())
    .dst_access_mask(vk::AccessFlags::empty())
    // We can erase the image_frame contents now, we already read them.
    .old_layout(vk::ImageLayout::UNDEFINED)
    // Transfer the layout back to general.
    .new_layout(vk::ImageLayout::GENERAL)
    .src_queue_family_index(queue_family_index)
    .dst_queue_family_index(vk::QUEUE_FAMILY_EXTERNAL)
    .image(image_frame)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });

device.cmd_pipeline_barrier(
    command_buffer_sampling,
    vk::PipelineStageFlags::TRANSFER,
    vk::PipelineStageFlags::BOTTOM_OF_PIPE,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*image_frame_memory_barrier],
);

device.end_command_buffer(command_buffer_sampling)?;

let semaphores = [semaphore];
let command_buffers = [command_buffer_sampling];
let submit_info = vk::SubmitInfo::builder()
    // Wait for our semaphore before starting.
    .wait_semaphores(&semaphores)
    .wait_dst_stage_mask(&[vk::PipelineStageFlags::ALL_COMMANDS])
    // Signal the semaphore after completion.
    .signal_semaphores(&semaphores)
    .command_buffers(&command_buffers);
device.queue_submit(queue, &[*submit_info], vk::Fence::null())?;

Here’s how the process looks now:

As you can see, I’m actually using another intermediate image rather than running the compute shader on image_frame directly. This is to handle the case when the game’s rendering framerate doesn’t match the video output framerate. There are two common scenarios which cause this to happen:

We’re doing motion blur via frame blending. The game runs at a higher FPS, e.g. output video FPS times 64, and each 64 captured frames are blended together to produce one output video frame. In this case every frame we’d accumulate pixel values of image_frame into the intermediate buffer, and only run color conversion once per the output video frame.
We’re recording a video of a tool-assisted speedrun (TAS). This is a script describing keyboard and mouse inputs to the game which can be played back deterministically, similarly to a demo file. There’s one important difference: a TAS script must set its own in-game FPS, so video recording (which wants constant output video FPS) needs to deal with it by duplicating or dropping output video frames, very much like regular screen recording software deals with varying FPS of a game it’s recording. In this case we don’t actually know for how long image_frame should be shown in the video until Half-Life renders the next frame, which means that we can only start the next processing stage right when we already need to capture the next frame. To add a bit of pipelining in this case, we save image_frame into the intermediate buffer right away, freeing the shared memory for the next capture, then process the intermediate buffer later when we know the duration.

Color Conversion #

I chose to output MP4 videos encoded in H.264 with YUV 4:2:0 pixel format and limited-range BT.709 color space. H.264 in an MP4 container with YUV 4:2:0 pixel format is playable on the most setups out of the currently available reasonable formats, while limited-range BT.709 should generally be selected as the color space by video players (for video resolutions that people will likely record), resulting in correct colors even if the video player ignores explicit color space tags.

Color space conversion is best done on the GPU as it’s a massively parallel operation (each pixel is essentially processed separately from the others), which GPUs are really good at. Thus, I implemented it with a compute shader:

#version 450
#extension GL_EXT_shader_8bit_storage: require

// I selected the value that gave the best performance on this workflow.
#define WORKGROUP_SIZE 4
layout (local_size_x = WORKGROUP_SIZE, local_size_y = WORKGROUP_SIZE, local_size_z = 1) in;

// Our intermediate buffer with the RGB image.
layout (binding = 0) uniform sampler2D image_sample;

// The output buffer in I420 format, ready to go into FFmpeg.
layout (binding = 1) buffer OutputBuffer {
    uint8_t output_buffer[];
};

void main() {
    vec2 size = textureSize(image_sample, 0);
    uint width = uint(size.x), height = uint(size.y);

    uint x = gl_GlobalInvocationID.x, y = gl_GlobalInvocationID.y;
    if (x >= width || y >= height)
        return;

    vec4 color = texture(image_sample, vec2(x + 0.5, y + 0.5));

    // The OpenGL coordinate system is upside-down compared to the usual video coordinate systems.
    // Let's invert it here in the shader.
    uint y_inv = height - y - 1;

    // Compute luminocity in accordance with the BT.709 specification.
    float Y = 16 + dot(color.rgb, vec3(0.2126, 0.7152, 0.0722) * (235 - 16));
    output_buffer[width * y_inv + x] = uint8_t(uint(round(Y)));

    // 4:2:0 subsampling means we compute one output color pixel
    // per each 2×2 square of input color pixels.
    if (x % 2 == 0 && y % 2 == 0) {
        // Subsample the color values assuming center chroma location.
        vec4 b = texture(image_sample, vec2(x + 1 + 0.5, y     + 0.5));
        vec4 c = texture(image_sample, vec2(x     + 0.5, y + 1 + 0.5));
        vec4 d = texture(image_sample, vec2(x + 1 + 0.5, y + 1 + 0.5));
        color = (color + b + c + d) / 4;

        // Convert color values in accordance with the BT.709 specification.
        float U = 128 + dot(color.rgb, vec3(-0.2126, -0.7152,  0.9278) / 1.8556 * (240 - 16));
        float V = 128 + dot(color.rgb, vec3( 0.7874, -0.7152, -0.0722) / 1.5748 * (240 - 16));

        // Write the values into the output buffer in the I420 format (planar YUV 4:2:0).
        output_buffer[width * height         + (width / 2) * (y_inv / 2) + (x / 2)] =
            uint8_t(uint(round(U)));
        output_buffer[width * height / 4 * 5 + (width / 2) * (y_inv / 2) + (x / 2)] =
            uint8_t(uint(round(V)));
    }
}

I’m using the VK_KHR_8bit_storage extension to write directly into a byte buffer from a shader. Without the extension I would have to make a regular image and then carefully write to the right pixel color values in the shader in such a way that, when interpreted as bytes, I’d get the correct I420 frame, but this would be very awkward. VK_KHR_8bit_storage is reasonably well supported so there are little compatibility concerns.

I get color values from the input frame, convert them to YUV with numeric constants from the specification and write the results out into the byte buffer in I420 order.

Let’s see how the input image and the output buffer are created.

// Intermediate image, input to the color conversion shader.
let create_info = vk::ImageCreateInfo {
    image_type: vk::ImageType::TYPE_2D,
    // Format suitable for frame blending, if I implement it in the future.
    format: vk::Format::R16G16B16A16_UNORM,
    extent: vk::Extent3D {
        width,
        height,
        depth: 1,
    },
    mip_levels: 1,
    array_layers: 1,
    samples: vk::SampleCountFlags::TYPE_1,
    tiling: vk::ImageTiling::OPTIMAL,
    usage: vk::ImageUsageFlags::STORAGE // For updating in-place during frame blending.
        | vk::ImageUsageFlags::TRANSFER_DST // For copying from image_frame.
        | vk::ImageUsageFlags::SAMPLED, // For reading from the color conversion shader.
    sharing_mode: vk::SharingMode::EXCLUSIVE,
    ..Default::default()
};
let image_sample = unsafe { device.create_image(&create_info, None)? };

// Allocate memory for the image.
let image_sample_memory_requirements =
    unsafe { device.get_image_memory_requirements(image_sample) };
let image_sample_memory_type_index = find_memorytype_index(
    &image_sample_memory_requirements,
    &memory_properties,
    vk::MemoryPropertyFlags::DEVICE_LOCAL,
)?;
let create_info = vk::MemoryAllocateInfo::builder()
    .allocation_size(image_sample_memory_requirements.size)
    .memory_type_index(image_sample_memory_type_index);
let image_sample_memory = unsafe { device.allocate_memory(&create_info, None)? };

// Bind them together.
unsafe { device.bind_image_memory(image_sample, image_sample_memory, 0)? };

To read from an image in a shader, we also need a sampler (which controls coordinate normalization and reading past the edge behavior) and an image view:

// Sampler.
let create_info = vk::SamplerCreateInfo::builder()
    // Reads past the edge return the edge pixel value.
    .address_mode_u(vk::SamplerAddressMode::CLAMP_TO_EDGE)
    .address_mode_v(vk::SamplerAddressMode::CLAMP_TO_EDGE)
    .address_mode_w(vk::SamplerAddressMode::CLAMP_TO_EDGE)
    // We'll use coordinates from 0 to width/height, rather than from 0 to 1.
    .unnormalized_coordinates(true);
let sampler_sample = unsafe { device.create_sampler(&create_info, None)? };

// Image view.
let create_info = vk::ImageViewCreateInfo::builder()
    .image(image_sample)
    .view_type(vk::ImageViewType::TYPE_2D)
    .format(vk::Format::R16G16B16A16_UNORM)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });
let image_view_sample = unsafe { device.create_image_view(&create_info, None)? };

Now for the output buffer. Actually, we’ll need two buffers: one device-local buffer which the shader will write to directly, and one “staging” CPU-accessible buffer that we’ll copy the results to for reading into RAM.

This might seem strange: why use two buffers instead of one? As it turns out, CPU-accessible buffers can be really slow to use for shader output, making the two-buffer scheme orders of magnitude faster.

// Buffer for color conversion shader output.
let create_info = vk::BufferCreateInfo::builder()
    // 4:2:0 subsampling: full-resolution Y + quarter-resolution U, V.
    .size(width as u64 * height as u64 / 2 * 3)
    .usage(
        // For using as the shader output.
        vk::BufferUsageFlags::STORAGE_BUFFER
        // For using as a source of a copy operation into the CPU-accessible buffer.
        | vk::BufferUsageFlags::TRANSFER_SRC
    )
    .sharing_mode(vk::SharingMode::EXCLUSIVE);
let buffer_color_conversion_output = unsafe { device.create_buffer(&create_info, None)? };

let buffer_color_conversion_output_memory_requirements =
    unsafe { device.get_buffer_memory_requirements(buffer_color_conversion_output) };
let buffer_color_conversion_output_memory_type_index = find_memorytype_index(
    &buffer_color_conversion_output_memory_requirements,
    &memory_properties,
    // This buffer resides on the GPU.
    vk::MemoryPropertyFlags::DEVICE_LOCAL,
)?;
let create_info = vk::MemoryAllocateInfo::builder()
    .allocation_size(buffer_color_conversion_output_memory_requirements.size)
    .memory_type_index(buffer_color_conversion_output_memory_type_index);
let buffer_color_conversion_output_memory =
    unsafe { device.allocate_memory(&create_info, None)? };
unsafe {
    device.bind_buffer_memory(
        buffer_color_conversion_output,
        buffer_color_conversion_output_memory,
        0,
    )?
};

// CPU-accessible staging buffer.
let create_info = vk::BufferCreateInfo::builder()
    // Same size as the buffer above.
    .size(width as u64 * height as u64 / 2 * 3)
    // For using it as a destination of a copy operation.
    .usage(vk::BufferUsageFlags::TRANSFER_DST)
    .sharing_mode(vk::SharingMode::EXCLUSIVE);
let buffer = unsafe { device.create_buffer(&create_info, None)? };

let buffer_memory_requirements = unsafe { device.get_buffer_memory_requirements(buffer) };
let buffer_memory_type_index = find_memorytype_index(
    &buffer_memory_requirements,
    &memory_properties,
    // This buffer is CPU-accessible.
    vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_CACHED,
)?;
let create_info = vk::MemoryAllocateInfo::builder()
    .allocation_size(buffer_memory_requirements.size)
    .memory_type_index(buffer_memory_type_index);
let buffer_memory = unsafe { device.allocate_memory(&create_info, None)? };
unsafe { device.bind_buffer_memory(buffer, buffer_memory, 0)? };

To use all these objects in the shader we also need a descriptor set allocated from a descriptor pool with a descriptor set layout and filled with our sampler and buffer, as well as the shader itself, and a pipeline encompassing all of those objects, however all of that is fairly standard with nothing interesting to point out.

Let’s look at how the pipeline is launched.

let begin_info = vk::CommandBufferBeginInfo::builder()
    .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT);
device.begin_command_buffer(command_buffer_color_conversion, &begin_info)?;

// Set a barrier for the color conversion stage.
let image_sample_memory_barrier = vk::ImageMemoryBarrier::builder()
    // We were copying from image_frame to this intermediate image.
    .src_access_mask(vk::AccessFlags::TRANSFER_WRITE)
    // We will be reading from this image in a shader.
    .dst_access_mask(vk::AccessFlags::SHADER_READ)
    // The layouts reflect this too.
    .old_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
    .new_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .image(image_sample)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });

device.cmd_pipeline_barrier(
    command_buffer_color_conversion,
    vk::PipelineStageFlags::TRANSFER,
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*image_sample_memory_barrier],
);

// To run the color conversion shader, we need to bind its pipeline and descriptor sets.
device.cmd_bind_pipeline(
    command_buffer_color_conversion,
    vk::PipelineBindPoint::COMPUTE,
    pipeline,
);
device.cmd_bind_descriptor_sets(
    command_buffer_color_conversion,
    vk::PipelineBindPoint::COMPUTE,
    pipeline_layout,
    0,
    &[descriptor_set_color_conversion],
    &[],
);

// Run the shader.
device.cmd_dispatch(
    command_buffer_color_conversion,
    // Our workgroup size is 4, this subdivision reflects that.
    (width + 4 - 1) / 4,
    (height + 4 - 1) / 4,
    1,
);

// Barrier for the device-local buffer to copy it to the CPU-accessible buffer.
let buffer_memory_barrier = vk::BufferMemoryBarrier::builder()
    // We were writing to the device-local buffer in a shader.
    .src_access_mask(vk::AccessFlags::SHADER_WRITE)
    // We will be reading from it with a copy operation.
    .dst_access_mask(vk::AccessFlags::TRANSFER_READ)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .buffer(buffer_color_conversion_output)
    .offset(0)
    .size(vk::WHOLE_SIZE);

device.cmd_pipeline_barrier(
    command_buffer_color_conversion,
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::PipelineStageFlags::TRANSFER,
    vk::DependencyFlags::empty(),
    &[],
    &[*buffer_memory_barrier],
    &[],
);

// The copy command itself.
let buffer_copy = vk::BufferCopy::builder().size(width as u64 * height as u64 / 2 * 3);
device.cmd_copy_buffer(
    command_buffer_color_conversion,
    buffer_color_conversion_output,
    buffer,
    &[*buffer_copy],
);

// Barrier for the CPU-accessible buffer to read it from the host.
let buffer_memory_barrier = vk::BufferMemoryBarrier::builder()
    // We were copying into the buffer.
    .src_access_mask(vk::AccessFlags::TRANSFER_WRITE)
    // We will be reading it from the host.
    .dst_access_mask(vk::AccessFlags::HOST_READ)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .buffer(buffer)
    .offset(0)
    .size(vk::WHOLE_SIZE);

device.cmd_pipeline_barrier(
    command_buffer_color_conversion,
    vk::PipelineStageFlags::TRANSFER,
    vk::PipelineStageFlags::HOST,
    vk::DependencyFlags::empty(),
    &[],
    &[*buffer_memory_barrier],
    &[],
);

// While we're at it, also prepare the intermediate image for another iteration
// by transferring it back to the original TRANSFER_DST layout.
let image_sample_memory_barrier = vk::ImageMemoryBarrier::builder()
    .src_access_mask(vk::AccessFlags::empty())
    .dst_access_mask(vk::AccessFlags::TRANSFER_WRITE)
    // We don't care about the contents any more.
    .old_layout(vk::ImageLayout::UNDEFINED)
    .new_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
    .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED)
    .image(image_sample)
    .subresource_range(vk::ImageSubresourceRange {
        aspect_mask: vk::ImageAspectFlags::COLOR,
        base_mip_level: 0,
        level_count: 1,
        base_array_layer: 0,
        layer_count: 1,
    });
device.cmd_pipeline_barrier(
    command_buffer_color_conversion,
    vk::PipelineStageFlags::COMPUTE_SHADER,
    vk::PipelineStageFlags::TRANSFER,
    vk::DependencyFlags::empty(),
    &[],
    &[],
    &[*image_sample_memory_barrier],
);

device.end_command_buffer(command_buffer_color_conversion)?;

With the command buffer set up, we’re almost ready to send it to the GPU. However, this time we want to read the pixel values back on the CPU to send them to the video encoder, which means that we need to wait for the GPU to complete these operations. This is where a Vulkan fence comes into play: the GPU can signal a fence upon finishing a command buffer, and the CPU can wait for the fence. After that we’re free to map the CPU-accessible buffer and read from it as normal.

// To wait for GPU operation completion on the host, we need a fence.
let create_info = vk::FenceCreateInfo::default();
let fence = device.create_fence(&create_info, None)?;

let command_buffers = [command_buffer_color_conversion];
let submit_info = vk::SubmitInfo::builder().command_buffers(&command_buffers);
device.queue_submit(
    queue,
    &[*submit_info],
    // The fence will be signaled upon completion of the command buffer.
    fence,
)?;

// Wait for the fence to be signalled.
device.wait_for_fences(&[fence], true, u64::max_value())?;

// Map the CPU-accessible buffer to read from it normally.
let pixels = device.map_memory(
    buffer_memory,
    0,
    vk::WHOLE_SIZE,
    vk::MemoryMapFlags::empty(),
)?;

// Invalidate it to make sure we're reading up-to-date data.
let mapped_memory_range = vk::MappedMemoryRange::builder()
    .memory(buffer_memory)
    .size(vk::WHOLE_SIZE);
device.invalidate_mapped_memory_ranges(&[*mapped_memory_range])?;

{
    // Now we can treat it as a regular array of pixels.
    let pixels: &[u8] =
        slice::from_raw_parts(pixels.cast(), width as usize * height as usize / 2 * 3);

    // Send it to the encoder, possibly multiple times (for the TAS playback case).
    for _ in 0..frames {
        muxer.write_video_frame(pixels)?;
    }
}

// Unmap the memory and free the fence.
device.unmap_memory(buffer_memory);
device.destroy_fence(fence, None);

All in all, this is the simplified diagram of the complete video recording process:

Staging Buffer Memory Type #

For the staging buffer I went with the HOST_VISIBLE | HOST_CACHED memory type. HOST_VISIBLE is required to access it from the CPU, while HOST_CACHED brought a big performance boost. HOST_COHERENT, the third available flag, did not seem to bring any benefit.

Let’s look at a performance visualization of several ways of reading memory from the GPU. The measurements were done recording at the 2560×1440 resolution and 60 FPS.

First, no staging buffer. The shader writes directly to a CPU-accessible buffer:

The green rectangle is waiting for the Vulkan fence, while the rest (not filled on this particular trace) is reading from the buffer and writing it to the pipe. The whole process takes 107 ms which is… quite underwhelming. This adds up to about 0.16× real-time speed of capturing.

Next, I’ll add a staging buffer, but without the HOST_CACHED flag. I didn’t have the next performance traces left from the time when I did the initial measurements, so I captured new ones on the latest version, where color conversion and muxing live on the second thread, but otherwise nothing changed in this function.

20 ms, much better! The process got more than five times faster. Note how the time waiting for the fence (so GPU work duration) went down too: the GPU can write to a device-local buffer much faster.

Finally, let’s add HOST_CACHED:

Just 3.4 ms, almost six times faster than before, wow! This time the improvement comes mainly from reading the buffer on the CPU side with no change to the GPU work duration. And we’re definitely exceeding the real-time encoding threshold here, by a factor of more than four.

The key takeaways are to use an extra staging buffer instead of writing from a shader directly to a CPU-accessible buffer and use HOST_CACHED for much faster memory reading.

Video Encoding #

For video encoding I use FFmpeg by running the ffmpeg binary and sending it content through a pipe. A more natural solution might seem to use the FFmpeg API as a library (in fact, I did just that in hl-capture), however this option has some drawbacks:

When using FFmpeg as a library, video encoding happens in-process. Since Half-Life is a 32-bit application, this means the RAM is heavily limited. In practice, 4K recording in hl-capture already starts hitting out of memory situations, and higher resolutions are straight up impossible. One of the goals for the new video recording was to lift this restriction.
The FFmpeg API has an over a decade long history, which really shows. It’s full of different ways of doing the same thing, each filled with gotchas and compatibility traps. The best way of working with FFmpeg seems to be to delegate all this trouble to the official binaries, which have at least some hope of working properly in most situations.
The Rust FFmpeg wrapper have not seen updates since 2015, although it seems that recently a few new crates have appeared to fill that gap; I haven’t looked at them thoroughly though.

Thus, I spawn ffmpeg with the correct arguments (notably, specifying the color space parameters) and pipe video and audio data to it, manually muxed into the NUT container format. The ffmpeg command line I’m using looks like this:

ffmpeg -f nut             # Input format.
       -i pipe:           # Data comes on the standard input.
       -c:v libx264       # Use libx264 for H.264 encoding.
       -crf 15            # For all intents and purposes, CRF 15 is visually lossless.
       -preset ultrafast  # Speed is a priority to avoid bottlenecks on encoding.

       # We're using BT.709…
       -color_primaries bt709
       -color_trc bt709
       -colorspace bt709
       # …with limited color range…
       -color_range tv
       # …and center chroma location.
       -chroma_sample_location center

       # Move the MP4 index to the start of the file for fast playback start when streaming.
       -movflags +faststart
       # Overwrite the file if it exists.
       -y
       output_filename.mp4

Conclusion #

The new Half-Life video recording works well on Linux and Windows and across a wide range of GPUs. Thanks to running FFmpeg out of process, it’s easy to record at very high resolutions such as 8K and possibly even 16K, all while running extremely fast thanks to GPU color conversion and a pipelined architecture.

To sum up, here are the main factors contributing to the performance:

Running most of the processing on a second thread so it runs parallel to the game working on the next frame.
Converting the color space from RGB to YUV on the GPU which is extremely well suited for this sort of task.
Using a HOST_CACHED staging buffer for best performance of getting data from the GPU to the CPU.
Encoding with libx264 using the ultrafast preset to avoid encoding bottlenecks.

A very interesting potential improvement is to do encoding on the GPU. libx264, even if very fast, is still a software encoder; besides, transferring compressed bitstream from the GPU should be faster than transferring full uncompressed YUV 4:2:0 frames. I’m looking forward to the Vulkan video extensions proposal which exposes GPU decoding and encoding through the Vulkan API, which I’m conveniently already using.

Even if the current encoding speed is “fast enough”, GPU encoding might allow using newer video compression formats by bringing the encoding performance on par with libx264.

Finally, if you’re doing any sort of Vulkan development, I heavily recommend using the validation layers as they automatically catch many mistakes and subtle errors in Vulkan usage.

Average day writing Vulkan on AMD

I did most of the development and optimization on my machine with an Intel i7-3770k CPU (overclocked to 4.2 GHz) and an AMD RX 580 GPU. ↩︎

GSoC 2018: Overview

Fri, 10 Aug 2018 13:08:36 +0000

Introduction #

Throughout the summer I was working on librsvg, a GNOME library for rendering SVG files to Cairo surfaces. This post is an overview of the work I did with relevant links.

My Results #

For the project I was to port the SVG filter infrastructure of librsvg from C to Rust, adding all missing filter tests from the SVG test suite along the way. I was also expected to implement abstractions to make the filter implementation more convenient, including Rust iterators over the surface pixels. Here’s a list of all merge requests accepted into librsvg as part of my GSoC project:

Splitting the existing C filter code into individual files for each filter:
- merge request.
Porting of the filter infrastructure to Rust, fixing bugs, implementing missing features and adding tests:
Improving performance of filters:
Miscellaneous fixes:
- enabling debuginfo in release builds,
- distcheck fix.

Here’s a convenient link to see all of these merge requests in GitLab: https://gitlab.gnome.org/GNOME/librsvg/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=YaLTeR&label_name[]=GSoC%202018

All of this code was accepted into the mainline and will appear in the next stable release of librsvg. I also wrote the following blog posts detailing some interesting things I worked on as part of the GSoC project:

GSoC 2018: Filter Infrastructure talking about the filter infrastructure, pixel iterators and benchmarking and showing an example of implementing a filter primitive;
GSoC 2018: Safe Shared Access to Cairo Image Surfaces showcasing the abstractions for safe shared access to Cairo image surface pixel data and showing how it can enable additional optimizations on the example of the alpha-only optimizations;
GSoC 2018: Parallelizing Filters with Rayon covering the parallelization of two computationally-intensive filters using the rayon crate.

Further Work #

There are a couple of fixes which still need to be done for filters to be feature-complete:

Fixing filters operating on off-screen nodes. Currently all intermediate surfaces are limited to the original SVG view area so anything off-screen is inaccessible to filters even when it should be. This is blocked on some considerable refactoring in the main librsvg node drawing code which is currently underway.
Implementing the filterRes property. This property allows to set the pixel resolution for filter operations and is one of the ways of achieving more resolution-independent rendering results. While it can be implemented with the current code as is, it will be much more convenient to account for it while refactoring the code to fix the previous issue.
Implementing the enable-background property. The BackgroundImage filter input should adhere to this property when picking which nodes to include in the background image, whereas it currently doesn’t.

GSoC 2018: Parallelizing Filters with Rayon

Fri, 10 Aug 2018 12:08:10 +0000

Introduction #

I’m working on SVG filter effects in librsvg, a GNOME library for rendering SVG files to Cairo surfaces. After finishing porting all filters from C to Rust and adding tests, I started investigating the filter performance. With the codebase converted to Rust, I am able to confidently apply important optimizations such as parallelization. In this post I’ll show how I parallelized two computation-intensive filter primitives.

Rayon #

Rayon is a Rust crate for introducing parallelism into existing code. It utilizes Rust’s type system to guarantee memory safety and data-race free execution. Rayon mimics the standard Rust iterators, so it’s possible to convert the existing iterator-using code with next to no changes. Compare the following single-threaded and parallel code:

// Single-threaded.
fn sum_of_squares(input: &[i32]) -> i32 {
    input.iter()
         .map(|i| i * i)
         .sum()
}

// Parallelized with rayon.
use rayon::prelude::*;
fn sum_of_squares(input: &[i32]) -> i32 {
    input.par_iter()
         .map(|i| i * i)
         .sum()
}

By merely using .par_iter() instead of .iter(), the computation becomes parallelized.

Parallelizing Lighting Filters #

Going forward with analyzing and improving the performance of the infamous mobile phone case, the two biggest time sinks were the lighting and the Gaussian blur filter primitives. It can be easily seen on the callgrind graph from KCachegrind:

Since I was working on optimizing the lighting filters, I decided to try out parallelization there first. The lighting filter primitives in SVG (feDiffuseLighting and feSpecularLighting) can be used to cast light onto the canvas using a render of an existing SVG node as a bump map. The computation is quite involved, but it boils down to performing a number of arithmetic operations for each pixel of the input surface independently of the others—a perfect target for parallelization. This is what the code initially looked like:

let mut output_surface = ImageSurface::create(
    cairo::Format::ARgb32,
    input_surface.width(),
    input_surface.height(),
)?;

let output_stride = output_surface.get_stride() as usize;
let mut output_data = output_surface.get_data().unwrap();

let mut compute_output_pixel = |x, y, normal: Normal| {
    let output_pixel = /* expensive computations */;

    output_data.set_pixel(output_stride, output_pixel, x, y);
};

// Compute the edge pixels
// <...>

// Compute the interior pixels
for y in bounds.y0 as u32 + 1..bounds.y1 as u32 - 1 {
    for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
        compute_output_pixel(
            x,
            y,
            interior_normal(&input_surface, bounds, x, y),
        );
    }
}

The edge pixel computation is separated out for optimization reasons and it’s not important. We want to focus on the main loop over the interior pixels: it takes up the most time. What we’d like to do is to take the outer loop over the image rows and run it in parallel on a thread pool. Since each row (well, each pixel in this case) is computed independently of the others, we should be able to do it without much hassle. However, we cannot do it right away: the compute_output_pixel closure mutably borrows output_data, so sharing it over multiple threads would mean multiple threads get concurrent mutable access to all image pixels, which could result in a data race and so won’t pass the borrow checker of the Rust compiler. Instead, we can split the output_data slice into row-sized non-overlapping chunks and feed each thread only those chunks that it needs to process. This way no thread can access the data of another thread. Let’s change the closure to accept the target slice (as opposed to borrowing it from the enclosing scope). Since the slices will start from the beginning of each row rather than the beginning of the whole image, we’ll also add an additional base_y argument to correct the offsets.

let compute_output_pixel =
    |mut output_slice: &mut [u8],
     base_y,
     x,
     y,
     normal: Normal| {
        let output_pixel = /* expensive computations */;

        output_slice.set_pixel(
            output_stride,
            output_pixel,
            x,
            y - base_y,
        );
    };

// Compute the interior pixels
for y in bounds.y0 as u32 + 1..bounds.y1 as u32 - 1 {
    for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
        compute_output_pixel(
            output_data,
            0,
            x,
            y,
            interior_normal(&input_surface, bounds, x, y),
        );
    }
}

Now we can convert the outer loop to operate through iterators using the chunks_mut() method of a slice which does exactly what we want: returns the slice in evenly sized non-overlapping mutable chunks.

let compute_output_pixel =
    |mut output_slice: &mut [u8],
     base_y,
     x,
     y,
     normal: Normal| {
        let output_pixel = /* expensive computations */;

        output_slice.set_pixel(
            output_stride,
            output_pixel,
            x,
            y - base_y,
        );
    };

// Compute the interior pixels
let first_row = bounds.y0 as u32 + 1;
let one_past_last_row = bounds.y1 as u32 - 1;
let first_pixel = (first_row as usize) * output_stride;
let one_past_last_pixel =
    (one_past_last_row as usize) * output_stride;

output_data[first_pixel..one_past_last_pixel]
    .chunks_mut(output_stride)
    .zip(first_row..one_past_last_row)
    .for_each(|(slice, y)| {
        for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
            compute_output_pixel(
                slice,
                y,
                x,
                y,
                interior_normal(
                    &input_surface,
                    bounds,
                    x,
                    y,
                ),
            );
        }
    });

And finally, parallelize by simply changing chunks_mut() to par_chunks_mut():

use rayon::prelude::*;

output_data[first_pixel..one_past_last_pixel]
    .par_chunks_mut(output_stride)
    .zip(first_row..one_past_last_row)
    .for_each(|(slice, y)| {
        for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
            compute_output_pixel(
                slice,
                y,
                x,
                y,
                interior_normal(
                    &input_surface,
                    bounds,
                    x,
                    y,
                ),
            );
        }
    });

Let’s see if the parallelization worked! Here I’m using time to measure how long it takes to render the mobile phone SVG. Before parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
6.95user 0.66system **0:07.62**elapsed **99%**CPU (0avgtext+0avgdata 270904maxresident)k
0inputs+2432outputs (0major+714373minor)pagefaults 0swaps

After parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
7.47user 0.63system **0:06.04**elapsed **134%**CPU (0avgtext+0avgdata 271328maxresident)k
0inputs+2432outputs (0major+714460minor)pagefaults 0swaps

Note that even though the user time went up, the elapsed time went down by 1.5 seconds, and the CPU utilization increased past 100%. Success!

Parallelizing Gaussian Blur #

Next, I set out to parallelize Gaussian blur, the biggest timesink for the phone and arguably one of the most used SVG filters altogether. The SVG specification hints that for all reasonable values of the standard deviation parameter the blur can be implemented as three box blurs (taking the average value of the pixels) instead of the much more costly Gaussian kernel convolution. Pretty much every SVG rendering agent implements it this way since it’s much faster and librsvg is no exception. This is why you’ll see functions called box_blur and not gaussian_blur. Both box blur and Gaussian blur are separable convolutions, which means it’s possible to implement them as two passes, one of which is a loop blurring each row of the image and another is a loop blurring each column of the image independently of the others. For box blur specifically it allows for a much more optimized convolution implementation. In librsvg, the box blur function contains an outer loop over the rows or the columns of the input image, depending on the vertical argument and an inner loop over the columns or the rows, respectively. It uses i and j for the outer and inner loop indices and has some helper functions to convert those to the actual coordinates, depending on the direction.

// Helper functions for getting and setting the pixels.
let pixel = |i, j| {
    let (x, y) = if vertical { (i, j) } else { (j, i) };

    input_surface.get_pixel_or_transparent(bounds, x, y)
};

let mut set_pixel = |i, j, pixel| {
    let (x, y) = if vertical { (i, j) } else { (j, i) };

    output_data.set_pixel(output_stride, pixel, x, y);
};

// Main loop
for i in other_axis_min..other_axis_max {
    // Processing the first pixel
    // <...>

    // Inner loop
    for j in main_axis_min + 1..main_axis_max {
        // <...>
    }
}

Trying to convert this code to use chunks_mut() just like the lighting filters, we stumble on an issue: if the outer loop is iterating over columns, rather than rows, the output slices for all individual columns overlap (because the pixels are stored in row-major order). We need some abstraction, like a matrix slice, that can be split into non-overlapping mutable subslices by rows or by columns. The first thing that comes to mind is to try using the Matrix type from the nalgebra crate which does have that functionality. However, it turns out that nalgebra doesn’t currently support rayon or even have by-row or by-column iterators. I tried implementing my own iterators but that required some very non-obvious unsafe code with odd trait bound restrictions which I really wasn’t sure were correct. Thus, I scrapped that code and made my own wrapper for the ImageSurface which only contains things needed for this particular use case. To reiterate, we need a wrapper that:

provides write access to the image pixels,
can be split by row or by column into non-overlapping chunks,
is Send, i.e. can be safely sent between threads (for parallelizing).

Here’s what I came up with:

struct UnsafeSendPixelData<'a> {
    width: u32,
    height: u32,
    stride: isize,
    ptr: NonNull<u8>,
    _marker: PhantomData<&'a mut ()>,
}

unsafe impl<'a> Send for UnsafeSendPixelData<'a> {}

impl<'a> UnsafeSendPixelData<'a> {
    /// Creates a new `UnsafeSendPixelData`.
    ///
    /// # Safety
    /// You must call `cairo_surface_mark_dirty()` on the
    /// surface once all instances of `UnsafeSendPixelData`
    /// are dropped to make sure the pixel changes are
    /// committed to Cairo.
    #[inline]
    unsafe fn new(
        surface: &mut cairo::ImageSurface,
    ) -> Self {
        assert_eq!(
            surface.get_format(),
            cairo::Format::ARgb32
        );
        let ptr = surface.get_data().unwrap().as_mut_ptr();

        Self {
            width: surface.get_width() as u32,
            height: surface.get_height() as u32,
            stride: surface.get_stride() as isize,
            ptr: NonNull::new(ptr).unwrap(),
            _marker: PhantomData,
        }
    }

    /// Sets a pixel value at the given coordinates.
    #[inline]
    fn set_pixel(&mut self, pixel: Pixel, x: u32, y: u32) {
        assert!(x < self.width);
        assert!(y < self.height);

        let value = pixel.to_u32();

        unsafe {
            let ptr = self.ptr.as_ptr().offset(
                y as isize * self.stride + x as isize * 4,
            ) as *mut u32;
            *ptr = value;
        }
    }

    /// Splits this `UnsafeSendPixelData` into two at the
    /// given row.
    ///
    /// The first one contains rows `0..index` (index not
    /// included) and the second one contains rows
    /// `index..height`.
    #[inline]
    fn split_at_row(self, index: u32) -> (Self, Self) {
        assert!(index <= self.height);

        (
            UnsafeSendPixelData {
                width: self.width,
                height: index,
                stride: self.stride,
                ptr: self.ptr,
                _marker: PhantomData,
            },
            UnsafeSendPixelData {
                width: self.width,
                height: self.height - index,
                stride: self.stride,
                ptr: NonNull::new(unsafe {
                    self.ptr
                        .as_ptr()
                        .offset(index as isize * self.stride)
                }).unwrap(),
                _marker: PhantomData,
            },
        )
    }

    /// Splits this `UnsafeSendPixelData` into two at the
    /// given column.
    ///
    /// The first one contains columns `0..index` (index not
    /// included) and the second one contains columns
    /// `index..width`.
    #[inline]
    fn split_at_column(self, index: u32) -> (Self, Self) {
        assert!(index <= self.width);

        (
            UnsafeSendPixelData {
                width: index,
                height: self.height,
                stride: self.stride,
                ptr: self.ptr,
                _marker: PhantomData,
            },
            UnsafeSendPixelData {
                width: self.width - index,
                height: self.height,
                stride: self.stride,
                ptr: NonNull::new(unsafe {
                    self.ptr
                        .as_ptr()
                        .offset(index as isize * 4)
                }).unwrap(),
                _marker: PhantomData,
            },
        )
    }
}

The wrapper contains a pointer to the data rather than a mutable slice of the data, so the intermediate pixels (which cannot be accessed through set_pixel()) are not mutably aliased between different instances of UnsafeSendPixelData. Now it’s possible to implement an iterator over the rows or the columns for this wrapper, however I went with a different, simpler approach: using rayon’s scope functionality which allows spawning worker threads directly into rayon’s thread pool. First, let’s change the existing code to operate on individual rows or columns, just like we did with the lighting filters:

// The following loop assumes the first row or column of
// `output_data` is the first row or column inside `bounds`.
let mut output_data = if vertical {
    output_data.split_at_column(bounds.x0 as u32).1
} else {
    output_data.split_at_row(bounds.y0 as u32).1
};

for i in other_axis_min..other_axis_max {
    // Split off one row or column and launch its processing
    // on another thread. Thanks to the initial split before
    // the loop, there's no special case for the very first
    // split.
    let (mut current, remaining) = if vertical {
        output_data.split_at_column(1)
    } else {
        output_data.split_at_row(1)
    };

    output_data = remaining;

    // Helper function for setting the pixels.
    let mut set_pixel = |j, pixel| {
        // We're processing rows or columns one-by-one, so
        // the other coordinate is always 0.
        let (x, y) = if vertical { (0, j) } else { (j, 0) };
        current.set_pixel(pixel, x, y);
    };

    // Processing the first pixel
    // <...>

    // Inner loop
    for j in main_axis_min + 1..main_axis_max {
        // <...>
    }
}

I could avoid the current and the base_i arguments to the set_pixel closure because I can declare the closure from within the loop, whereas in the lighting filters code the compute_output_pixel closure had to be used and so declared outside of the main loop. Now it’s a simple change to split the work across rayon’s threads:

// The following loop assumes the first row or column of
// `output_data` is the first row or column inside `bounds`.
let mut output_data = if vertical {
    output_data.split_at_column(bounds.x0 as u32).1
} else {
    output_data.split_at_row(bounds.y0 as u32).1
};

// Establish a scope for the threads.
rayon::scope(|s| {
    for i in other_axis_min..other_axis_max {
        // Split off one row or column and launch its
        // processing on another thread. Thanks to the
        // initial split before the loop, there's no special
        // case for the very first split.
        let (mut current, remaining) = if vertical {
            output_data.split_at_column(1)
        } else {
            output_data.split_at_row(1)
        };

        output_data = remaining;

        // Spawn the thread for this row or column.
        s.spawn(move |_| {
            // Helper function for setting the pixels.
            let mut set_pixel = |j, pixel| {
                // We're processing rows or columns
                // one-by-one, so the other coordinate is
                // always 0.
                let (x, y) =
                    if vertical { (0, j) } else { (j, 0) };
                current.set_pixel(pixel, x, y);
            };

            // Processing the first pixel
            // <...>

            // Inner loop
            for j in main_axis_min + 1..main_axis_max {
                // <...>
            }
        });
    }
});

Let’s measure the performance. In the end of the previous section we had:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
7.47user 0.63system **0:06.04**elapsed **134%**CPU (0avgtext+0avgdata 271328maxresident)k
0inputs+2432outputs (0major+714460minor)pagefaults 0swaps

And now after the parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
10.32user 1.10system **0:04.57**elapsed **250%**CPU (0avgtext+0avgdata 272588maxresident)k
0inputs+2432outputs (0major+1009498minor)pagefaults 0swaps

We cut another 1.5 seconds and further increased the CPU utilization!

Conclusion #

Rayon is an excellent crate which provides a multitude of ways for safely parallelizing Rust code. It builds on idiomatic concepts such as iterators, but also contains a number of other convenient ways for parallelizing code when standard iterators aren’t sufficient or wouldn’t be very convenient. It uses the Rust’s type system to statically guarantee the absence of data races and manages the low-level details on its own allowing the programmer to focus on the actual computation. The parallelized filters are now included in the latest development release of librsvg for testing if using multiple threads leads to any issues downstream.

GSoC 2018: Safe Shared Access to Cairo Image Surfaces

Mon, 16 Jul 2018 09:53:00 +0000

Introduction #

I’m working on librsvg, a GNOME SVG rendering library, to port the SVG filter effects and related infrastructure from C to Rust. Librsvg uses Cairo, a 2D graphics library, for most of its drawing operations. Cairo can draw to a number of different surfaces like XCB and Xlib windows and pixmaps, PDF documents and PostScript files.

Image Surfaces #

Filter effects operate on rasterized bitmaps, so most of them need direct access to pixel data. There’s a special Cairo surface type for that: an image surface, backed by a memory buffer. Filters receive their input images in image surfaces, perform the necessary pixel operations and return new image surfaces with the results. In the Rust bindings, image surfaces are represented by the ImageSurface struct. It has a get_data() method which returns an ImageSurfaceData, which in turn acts as a slice into the underlying memory. Since ImageSurfaceData gives mutable access to the surface memory, it must ensure there are no other views into the same memory to comply with Rust’s ownership rules. This is achieved by checking that the ImageSurface reference count is equal to 1 in get_data(), which means that no other references to the ImageSurface exist. Furthermore, the ImageSurface is borrowed mutably for the lifetime of ImageSurfaceData. This is needed to prevent cloning the surface after obtaining ImageSurfaceData and subsequently drawing on it with a Cairo context while the view into the surface memory still exists. While this scheme does work, it offers only mutable, unique access to the pixel data. In the filter code, it’s much more convenient to have multiple references to the input surfaces with read-only memory access. Simply adding a struct similar to ImageSurfaceData which provides only a read-only view into the memory does not allow to drop the unique reference constraint, because it’s always possible to use the other reference to concurrently mutate the pixel data.

Shared Image Surface #

To work around the constraints, I ended up creating a special wrapper for ImageSurfaces:

#[derive(Debug, Clone)]
struct SharedImageSurface {
    surface: ImageSurface,

    data_ptr: NonNull<u8>, // *const.
    width: i32,
    height: i32,
    stride: isize,
}

The idea is to wrap a unique ImageSurface and provide just read-only access to it, while allowing cloning of SharedImageSurface itself. This way there can be multiple SharedImageSurfaces without compromising soundness because without direct access to the underlying ImageSurface it’s impossible to mutate it in any way from the outside. Additionally, since we know the surface won’t be modified, we can cache some common properties to get rid of extra C calls which can’t be easily optimized away. The SharedImageSurface constructor ensures the ImageSurface it receives is unique:

impl SharedImageSurface {
    pub fn new(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        // get_pixel() assumes ARgb32.
        assert_eq!(
            surface.get_format(),
            cairo::Format::ARgb32
        );

        // Ensure the access is unique.
        let reference_count = unsafe {
            cairo_sys::cairo_surface_get_reference_count(
                surface.to_raw_none(),
            )
        };
        assert_eq!(reference_count, 1);

        // Flush any pending drawing operations before
        // accessing the memory directly.
        surface.flush();
        if surface.status() != cairo::Status::Success {
            return Err(surface.status());
        }

        let data_ptr = NonNull::new(unsafe {
            cairo_sys::cairo_image_surface_get_data(
                surface.to_raw_none(),
            )
        }).unwrap();

        let width = surface.get_width();
        let height = surface.get_height();
        let stride = surface.get_stride() as isize;

        Ok(Self {
            surface,
            data_ptr,
            width,
            height,
            stride,
        })
    }
}

And other methods on SharedImageSurface provide access to various surface properties, as well as the pixel data:

impl SharedImageSurface {
    /// Returns the surface width.
    #[inline]
    pub fn width(&self) -> i32 {
        self.width
    }

    /// Returns the surface height.
    #[inline]
    pub fn height(&self) -> i32 {
        self.height
    }

    /// Retrieves the pixel value at the given coordinates.
    #[inline]
    pub fn get_pixel(&self, x: u32, y: u32) -> Pixel {
        assert!(x < self.width as u32);
        assert!(y < self.height as u32);

        let offset =
            y as isize * self.stride + x as isize * 4;
        let ptr = self.data_ptr.as_ptr().offset(offset);

        // According to Cairo documentation, the pixel values
        // for the ARgb32 format should be read using a
        // platform-native u32.
        let value = unsafe { *(ptr as *const u32) };

        Pixel {
            r: ((value >> 16) & 0xFF) as u8,
            g: ((value >> 8) & 0xFF) as u8,
            b: (value & 0xFF) as u8,
            a: ((value >> 24) & 0xFF) as u8,
        }
    }
}

Another nice property of wrapping ImageSurfaces is that it’s possible to provide extra utility methods for working with pixel data. Right now we have methods for extracting the alpha channel, resizing the surface, doing linear sRGB to sRGB and vice versa conversions and performing convolutions useful for primitives like Gaussian blur. Pixel iterators I showcased in the previous post were switched to take SharedImageSurfaces as input. One downside is that any other Cairo APIs that take surfaces as a read-only input need to be wrapped too. Fortunately, so far we needed only the Context::set_source_surface() method which allows using the given surface as the pixel source for drawing operations:

impl SharedImageSurface {
    /// Calls `set_source_surface()` on the given Cairo
    /// context.
    #[inline]
    pub fn set_as_source_surface(
        &self,
        cr: &cairo::Context,
        x: f64,
        y: f64,
    ) {
        cr.set_source_surface(&self.surface, x, y);
    }
}

Converting a SharedImageSurface back into a regular ImageSurface to return it from the filter code does a reference count check too: if it isn’t equal to 1, there are other SharedImageSurfaces pointing at the same ImageSurface, which means we cannot simply return the surface as it can be used for modifying the pixel data thus breaking the SharedImageSurface invariants. In this case a copy of the surface is created and returned:

impl SharedImageSurface {
    /// Converts this `SharedImageSurface` back into a Cairo
    /// image surface.
    #[inline]
    pub fn into_image_surface(
        self,
    ) -> Result<ImageSurface, cairo::Status> {
        let reference_count = unsafe {
            cairo_sys::cairo_surface_get_reference_count(
                self.surface.to_raw_none(),
            )
        };

        if reference_count == 1 {
            Ok(self.surface)
        } else {
            // If there are any other references, copy the
            // underlying surface.
            let bounds = IRect {
                x0: 0,
                y0: 0,
                x1: self.width,
                y1: self.height,
            };

            self.copy_surface(bounds)
        }
    }
}

Alpha-Only Optimizations #

The most heavy filter primitives are the ones doing image convolutions, mainly the Gaussian blur primitive. A convolution involves computing a weighted sum of pixels within a rectangle for every input pixel. The Gaussian blur primitive is frequently used for creating shadows for other elements. In this case it takes the rasterized element’s alpha channel as an input and blurs it. Then the blurred surface is offset a little and drawn below the input surface to create a shadow.

When a convolution is applied to an alpha-only image, there’s no need to compute the weighted sum for the other three color channels. However, going through the whole input image to check if it only contains meaningful data in the alpha channel is rather costly. Thankfully, we can avoid that step altogether by caching this property. I added a field into SharedImageSurface to indicate whether the current surface is alpha-only, along with a special constructor:

struct SharedImageSurface {
    /* ... */

    /// Whether this surface contains meaningful data only
    /// in the alpha channel.
    ///
    /// This is used for optimizations, particularly in
    /// `convolve()` to skip processing other channels.
    alpha_only: bool,
}

impl SharedImageSurface {
    pub fn new(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        /* ... */

        Ok(Self {
            surface,
            data_ptr,
            width,
            height,
            stride,
            // Default to not alpha only.
            alpha_only: false,
        })
    }

    /// Creates a `SharedImageSurface` from a unique
    /// `ImageSurface` with meaningful data only in the alpha
    /// channel.
    #[inline]
    pub fn new_alpha_only(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        let mut rv = Self::new(surface)?;
        rv.alpha_only = true;
        Ok(rv)
    }
}

This constructor is used automatically in the extract_alpha() method used to separate the alpha channel of the input surface:

impl SharedImageSurface {
    /// Returns a surface with black background and alpha
    /// channel matching this surface.
    pub fn extract_alpha(
        &self,
        bounds: IRect,
    ) -> Result<SharedImageSurface, cairo::Status> {
        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let output_stride =
            output_surface.get_stride() as usize;
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            for (x, y, Pixel { a, .. }) in
                Pixels::new(self, bounds)
            {
                let output_pixel = Pixel {
                    r: 0,
                    g: 0,
                    b: 0,
                    a,
                };

                output_data.set_pixel(
                    output_stride,
                    output_pixel,
                    x,
                    y,
                );
            }
        }

        // The returned surface is alpha-only!
        SharedImageSurface::new_alpha_only(output_surface)
    }
}

Finally, the pixel processing methods check the alpha-only flag to reduce the number of operations or skip unneeded processing altogether:

impl SharedImageSurface {
    /// Returns a surface with pre-multiplication of color
    /// values undone.
    pub fn unpremultiply(
        &self,
        bounds: IRect,
    ) -> Result<SharedImageSurface, cairo::Status> {
        // Unpremultiplication doesn't affect the alpha
        // channel.
        if self.alpha_only {
            return Ok(self.clone());
        }

        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let stride = output_surface.get_stride() as usize;
        {
            let mut data =
                output_surface.get_data().unwrap();

            for (x, y, pixel) in Pixels::new(self, bounds) {
                data.set_pixel(
                    stride,
                    pixel.unpremultiply(),
                    x,
                    y,
                );
            }
        }

        SharedImageSurface::new(output_surface)
    }

    /// Performs a convolution.
    pub fn convolve(
        &self,
        bounds: IRect,
        target: (i32, i32),
        kernel: &Matrix<f64>,
        edge_mode: EdgeMode,
    ) -> Result<SharedImageSurface, cairo::Status> {
        assert!(kernel.rows() >= 1);
        assert!(kernel.cols() >= 1);

        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let output_stride =
            output_surface.get_stride() as usize;
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            if self.alpha_only {
                // Perform a convolution, taking a weighted
                // sum of only the alpha channel.
            } else {
                // Perform a convolution, taking a weighted
                // sum of all four color channels.
            }
        }

        if self.alpha_only {
            SharedImageSurface::new_alpha_only(
                output_surface,
            )
        } else {
            SharedImageSurface::new(output_surface)
        }
    }
}

Validating Performance #

The main motivating example for the optimizations is this old mobile phone SVG file from a 9-year-old issue on performance which takes about 40 seconds to render on my desktop PC:

To measure the performance impact, I built librsvg in release mode with debug info before doing alpha-only optimizations and used the perf tool:

perf record --call-graph=dwarf ./rsvg-convert -o /dev/null mobile_phone_01.svg

I then used the awesome FlameGraph tool to create a nice visualization of the data (click and open in a browser for an interactive SVG):

The large two-part column in the middle happens to be the Gaussian blur, taking up 50.63% of the total rendering time for the mobile phone. Turns out the blur operates on SourceAlpha, which contains just the alpha channel of the rasterized element the filter is applied to. After adding the alpha-only optimizations to convolution and other parts of the code like linear sRGB ⇔ sRGB conversion, the rendering time dropped from 40 seconds to 29 seconds, and the performance graph now looks like this:

The percentage of time taken by Gaussian blur dropped from 50.63% to 36.98%. You can also see a slight drop in the narrow column to the left of the Gaussian blur part: that’s the sRGB linearizations which became no-ops on those input images that were alpha-only.

Conclusion #

The project is coming along very well. I’ve just came back from GUADEC where I really pushed the filter rustification during the hacking days, porting all of the remaining filters from C to Rust. Now that all filters are in Rust and thoroughly tested, I’m looking at improving the performance, starting with the alpha-only optimization described in this post. Also since all SVG nodes are now in Rust (the filter primitives were the last ones), I was able to clean up the remaining C interop code for nodes! It’s great to see the intermediate FFI code gradually disappear as the entire subsystems get completely ported to Rust.

GSoC 2018: Filter Infrastructure

Fri, 08 Jun 2018 07:26:03 +0000

Introduction #

This summer I’m working on librsvg, a GNOME library for rendering SVG files, particularly on porting the SVG filter effects from C to Rust. That involves separating the code for different filters from one huge C file into individual files for each filter, and then porting the filter rendering infrastructure and the individual filters.

Thankfully, in the large C file the code for different filters was divided by comment blocks, so several vim macros later I was done with the not so exciting splitting part.

Representing Filters in Rust #

SVG filter effects are applied to an existing SVG element to produce a modified graphical result. Each filter consists of a number of filter primitives. The primitives take raster images (bitmaps) as an input (this can be, for example, the rasterized element where the filter was applied, the background snapshot of the canvas at the time the filter was invoked, or an output of another filter primitive), do something with it (like move the pixels to a different position, apply Gaussian blur, or blend two input images together) and produce raster images as an output. Each filter primitive has a number of properties. The common properties include the bounds of the region where the filter primitive is doing its processing, the name assigned to the primitive’s result, and the input that the primitive operates on. I collected the common properties into the following types:

struct Primitive {
    x: Cell<Option<RsvgLength>>,
    y: Cell<Option<RsvgLength>>,
    width: Cell<Option<RsvgLength>>,
    height: Cell<Option<RsvgLength>>,
    result: RefCell<Option<String>>,
}

struct PrimitiveWithInput {
    base: Primitive,
    in_: RefCell<Option<Input>>,
}

Each filter primitive struct is meant to contain one of these two common types along with any extra properties as needed. The common types provide functions for parsing their respective properties so that code need not to be duplicated in each filter. Note that these properties are just “descriptions” of the final values to be used during rendering. For example, an RsvgLength can be equal to 2 or 50%, and the actual length in pixels is evaluated during rendering and depends on various rendering state such as the coordinate system in use and the size of the enclosing element. The filter primitive processing behavior is nicely described as a trait:

trait Filter {
    fn render(&self, ctx: &FilterContext)
        -> Result<FilterResult, FilterError>;
}

Here FilterContext contains various filter state such as the rasterized bitmap representation of the SVG element the filter is being applied to and results of previously rendered primitives, and allows retrieving the necessary input bitmaps. Successful rendering results in a FilterResult which has the name assigned to the primitive and the output image, and errors (like non-existent input filter primitive) end up in FilterError. When a filter is invoked, it goes through its child nodes (filter primitives) in order, render()s them and stores the results in the FilterContext.

Pixel Iteration #

Since many filter primitives operate on a per-pixel basis, it’s important to have a convenient way of transforming the pixel values. Librsvg uses image surfaces from Cairo, a 2D graphics library, for storing bitmaps. An image surface stores its pixel values in RGBA format in a large contiguous array row by row with optional strides between the rows. The plain way of accessing the values is image[y * stride + x * 4 + ch] where ch is 0, 1, 2 and 3 for R, G, B and A respectively. However, writing this out is rather tedious and error-prone. As the first step, I added a pixel value struct:

struct Pixel {
    pub r: u8,
    pub g: u8,
    pub b: u8,
    pub a: u8,
}

and extended cairo-rs’s image surface data accessor with the following methods:

fn get_pixel(
    &self,
    stride: usize,
    x: usize,
    y: usize,
) -> Pixel;

fn set_pixel(
    &mut self,
    stride: usize,
    pixel: Pixel,
    x: usize,
    y: usize,
);

using the known trick of declaring a trait containing the new methods and implementing it for the target type. Unfortunately, stride has to be passed through manually because the (foreign) data accessor type doesn’t offer a public way of retrieving it. Adding methods to cairo-rs directly would allow to get rid of this extra argument. Next, since the pattern of iterating over pixels of an image surface within the given bounds comes up rather frequently in filter primitives, I added a Pixels iterator inspired by the image crate. It allows writing code like this:

for (x, y, pixel) in Pixels::new(&image, bounds) {
    /* ... */
}

instead of the repetitive plain version:

for y in bounds.y0..bounds.y1 {
    for x in bounds.x0..bounds.x1 {
        let pixel = image.get_pixel(stride, x, y);
        /* ... */
    }
}

Filters with multiple input images can process pixels simultaneously in the following fashion using the standard Rust iterator combinators:

for (x, y, p, p2) in Pixels::new(&image, bounds)
    .map(|(x, y, p)| {
        (x, y, p, image2.get_pixel(stride, x, y))
    })
{
    let out_pixel = /* ... */;
    out_image.set_pixel(stride, out_pixel, x, y);
}

Benchmarking #

Rust is known for its zero-cost abstractions, however it’s still important to keep track of performance because it’s very well possible to write code in such a way that’s hard to optimize away. Fortunately, a benchmarking facility is provided on nightly Rust out of the box: the test feature with the Bencher type. Benchmark sources are usually placed in the benches/ subdirectory of the crate and look like this:

#![feature(test)]
extern crate rsvg_internals;

#[cfg(test)]
mod tests {
    use super::*;
    use test::Bencher;

    #[bench]
    fn my_benchmark_1(b: &mut Bencher) {
        /* initialization */

        b.iter(|| {
            /* code to be benchmarked */
        });
    }

    #[bench]
    fn my_benchmark_2(b: &mut Bencher) {
        /* ... */
    }

    /* ... */
}

After ensuring the crate’s crate-type includes "lib", you can run benchmarks with cargo +nightly bench. I created three benchmarks, one for the straightforward iteration:

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for y in BOUNDS.y0..BOUNDS.y1 {
        for x in BOUNDS.x0..BOUNDS.x1 {
            let base = y * stride + x * 4;

            r += image[base + 0] as usize;
            g += image[base + 1] as usize;
            b += image[base + 2] as usize;
            a += image[base + 3] as usize;
        }
    }

    (r, g, b, a)
})

One for iteration using get_pixel():

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for y in BOUNDS.y0..BOUNDS.y1 {
        for x in BOUNDS.x0..BOUNDS.x1 {
            let pixel = image.get_pixel(stride, x, y);

            r += pixel.r as usize;
            g += pixel.g as usize;
            b += pixel.b as usize;
            a += pixel.a as usize;
        }
    }

    (r, g, b, a)
})

And one for the Pixels iterator:

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for (_x, _y, pixel) in Pixels::new(&image, BOUNDS) {
        r += pixel.r as usize;
        g += pixel.g as usize;
        b += pixel.b as usize;
        a += pixel.a as usize;
    }

    (r, g, b, a)
})

Here are the results I’ve got:

test tests::bench_pixels                   ... bench:     991,137 ns/iter (+/- 62,654)
test tests::bench_straightforward          ... bench:     992,124 ns/iter (+/- 7,119)
test tests::bench_straightforward_getpixel ... bench:   1,034,037 ns/iter (+/- 11,121)

Looks like the abstractions didn’t introduce any overhead indeed!

Implementing a Filter Primitive #

Let’s look at how to write a simple filter primitive in Rust. As an example I’ll show the offset filter primitive which moves its input on the canvas by a specified number of pixels. Offset has an input and two additional properties for the offset amounts:

struct Offset {
    base: PrimitiveWithInput,
    dx: Cell<RsvgLength>,
    dy: Cell<RsvgLength>,
}

Since each filter primitive is an SVG node, it needs to implement NodeTrait which contains a function for parsing the node’s properties:

impl NodeTrait for Offset {
    fn set_atts(
        &self,
        node: &RsvgNode,
        handle: *const RsvgHandle,
        pbag: &PropertyBag,
    ) -> NodeResult {
        // Parse the common properties.
        self.base.set_atts(node, handle, pbag)?;

        // Parse offset-specific properties.
        for (_key, attr, value) in pbag.iter() {
            match attr {
                Attribute::Dx => self.dx.set(parse(
                    "dx",
                    value,
                    LengthDir::Horizontal,
                    None,
                )?),
                Attribute::Dy => self.dy.set(parse(
                    "dy",
                    value,
                    LengthDir::Vertical,
                    None,
                )?),
                _ => (),
            }
        }

        Ok(())
    }
}

Finally, we need to implement the Filter trait. Note that render() accepts an additional &RsvgNode argument, which refers to the filter primitive node. It’s different from &self in that it contains various common SVG node state.

impl Filter for Offset {
    fn render(
        &self,
        node: &RsvgNode,
        ctx: &FilterContext,
    ) -> Result<FilterResult, FilterError> {
        // Compute the processing region bounds.
        let bounds = self.base.get_bounds(ctx);

        // Compute the final property values.
        let cascaded = node.get_cascaded_values();
        let values = cascaded.get();

        let dx = self
            .dx
            .get()
            .normalize(&values, ctx.drawing_context());
        let dy = self
            .dy
            .get()
            .normalize(&values, ctx.drawing_context());

        // The final offsets depend on the currently active
        // affine transformation.
        let paffine = ctx.paffine();
        let ox = (paffine.xx * dx + paffine.xy * dy) as i32;
        let oy = (paffine.yx * dx + paffine.yy * dy) as i32;

        // Retrieve the input surface.
        let input_surface =
            get_surface(self.base.get_input(ctx))?;

        // input_bounds contains all pixels within bounds,
        // for which (x + ox) and (y + oy) also lie
        // within bounds.
        let input_bounds = IRect {
            x0: clamp(bounds.x0 - ox, bounds.x0, bounds.x1),
            y0: clamp(bounds.y0 - oy, bounds.y0, bounds.y1),
            x1: clamp(bounds.x1 - ox, bounds.x0, bounds.x1),
            y1: clamp(bounds.y1 - oy, bounds.y0, bounds.y1),
        };

        // Create an output surface.
        let mut output_surface =
            ImageSurface::create(
                cairo::Format::ARgb32,
                input_surface.get_width(),
                input_surface.get_height(),
            ).map_err(FilterError::OutputSurfaceCreation)?;

        let output_stride =
            output_surface.get_stride() as usize;

        // An extra scope is needed because output_data
        // borrows output_surface, but we need to move
        // out of it to return it.
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            for (x, y, pixel) in
                Pixels::new(&input_surface, input_bounds)
            {
                let output_x = (x as i32 + ox) as usize;
                let output_y = (y as i32 + oy) as usize;
                output_data.set_pixel(
                    output_stride,
                    pixel,
                    output_x,
                    output_y,
                );
            }
        }

        // Return the result of the processing.
        Ok(FilterResult {
            name: self.base.result.borrow().clone(),
            output: FilterOutput {
                surface: output_surface,
                bounds,
            },
        })
    }
}

Conclusion #

The project is coming along very nicely with a few simple filters already working in Rust and a couple of filter tests getting output closer to the reference images. I’ll be attending this year’s GUADEC, so I hope to see you there in July!

GSoC 2018: Introduction

Mon, 30 Apr 2018 18:43:40 +0000

Hello! #

I’m Ivan Molodetskikh, a student of the Moscow State University, and I love Rust. I learned about Rust after a couple of years of mainly programming personal projects in C++ (with a little bit of C#, Java and others here and there). Making my way through writing the first Rust project (what’s a better way to get familiar with a language than trying to make something interesting in it?) and trying out various features of the language, I frequently had moments of “This is actually so much better than the C++ alternative!”. Multiple times after inspecting a borrow-related error I realized this has bit me in the past in C++ in a form of a hard to find mistake. Borrowing, lifetimes, iterators, enums, Option and Result in particular are great once you get a feel of how to use them. The Rust community is amazing too with people always being positive and happy to help newcomers. Fast-forward two years, I have a couple of small Rust projects and some contributions and continuing to enjoy the language. So, it should be of no surprise that when I learned about GSoC I started looking for Rust-related projects. I applied to both Xi (a novel text editor with a fully async architecture) and librsvg (a GNOME library for rendering SVG files) and got accepted into librsvg on a project to help with the ongoing effort to port it to Rust, specifically the SVG filter effects.

The project #

Librsvg is a small library for rendering SVG files in the GNOME ecosystem. Its goal is to be a low-footprint library with a minimal API suitable for rendering things like icons and other images that appear on the desktop. Among other things SVG supports the so-called filter effects which generally (but not always) transform existing SVG elements in one way or another. For example, there’s a blur effect, an effect that moves its input by a fixed position, and an effect that loads an external raster image. Currently all filter effects in librsvg are implemented entirely in C. Additionally, most of them aren’t covered by specification conformance tests. For this project I’m going to be porting the filter infrastructure to Rust, adding all missing tests and making sure everything works correctly. There’s a number of things to get done as part of the project. I’ll start by separating the existing C filter code from one huge file into multiple small ones, one for each filter. This will make it easier to port filters to Rust one by one later. Next comes the most interesting part, experimenting with Rust abstractions over common filter actions, such as iterating over pixels in various ways, like one by one or using a square window. This has to be fast and ergonomic and support the different filter use cases. Finally, I’ll be porting all filter code from C to Rust and adding the missing filter tests along the way to make sure everything works right.

Conclusion #

I’ll be posting about my progress on this blog. If you want to contact me, I’m always in #rust on the GNOME IRC. And if you’re into video game speedruns, this summer I’ll be at the ESA hanging out and helping to commentate the Half-Life run. 🙂

Rust on Ivan Molodetskikh’s Webpage

Motion Blur for Half-Life Video Recording with Vulkan

How Motion Blur Works #

Frame Blending in bxt-rs #

Color Accumulation #

Recording Loop #

Synchronization #

Overhead #

Conclusion #

Appendix: Interactive Widget #

Fast Half-Life Video Recording with Vulkan

Specialized Recording Tools #

Prior Art #

Implementation #

Overview #

Choosing the Vulkan Crate #

Choosing the GPU #

Sharing Memory with OpenGL #

Capturing a Frame #

Color Conversion #

Staging Buffer Memory Type #

Video Encoding #

Conclusion #

GSoC 2018: Overview

Introduction #

My Results #

Further Work #

GSoC 2018: Parallelizing Filters with Rayon

Introduction #

Rayon #

Parallelizing Lighting Filters #

Parallelizing Gaussian Blur #

Conclusion #

GSoC 2018: Safe Shared Access to Cairo Image Surfaces

Introduction #

Image Surfaces #

Shared Image Surface #

Alpha-Only Optimizations #

Validating Performance #

Conclusion #

GSoC 2018: Filter Infrastructure

Introduction #

Representing Filters in Rust #

Pixel Iteration #

Benchmarking #

Implementing a Filter Primitive #

Conclusion #

GSoC 2018: Introduction

Hello! #

The project #

Conclusion #