Gnome on Ivan Molodetskikh’s Webpage

Just How Much Faster Are the GNOME 46 Terminals?

Sat, 06 Apr 2024 12:00:00 +0400

VTE (Virtual TErminal library) is the library underpinning various GNOME terminal emulators. It provides a GTK widget that shows a terminal view, which is used in apps like GNOME Terminal, Console, Black Box, Tilix, Terminator, Ptyxis, and others. It also powers embedded terminals in Builder and Workbench.

Over the GNOME 46 cycle, VTE has seen a lot of performance improvements. Christian Hergert mentioned some of them in his blog posts about VTE and about his work in GNOME 46. But how much did the performance actually improve? What should you, the user, expect to feel after installing a fresh Fedora 40 update and launching your favorite terminal?

Let’s measure and find out! If you don’t have time for measuring, you can skip straight to the finding out.

What Are We Measuring? #

There is no shortage of ways to define “performance”, especially when it comes to terminal emulators. One of the more tangible metrics is input latency. Roughly, it describes how quickly the program reacts to your actions: how much time passes from the moment you press a key on your keyboard to the change in color of the pixels on your monitor. Apps with low input latency feel snappy, whereas apps with high input latency can feel sluggish.

When the input latency is small-ish, you can get used to it and think it feels fine. However, comparing lower and higher input latency together (for example, by switching between two apps and typing in both) can make it quite noticeable. If you’ve ever heard people say they can’t go back to a 60 Hz monitor after trying out 144 Hz, that’s a similar effect (and input latency is partially responsible).

So, how do you measure it?

Measuring Input Latency #

There are tools like Typometer that measure the input latency in software by detecting key press events and recording the screen to detect a change in pixel color. This can work reasonably well but requires fiddling with your setup to make sure you’re not accidentally introducing any biases. For example, a screen capture API may return the new pixel colors a few milliseconds before or after they are shown on the monitor, depending on the system setup, and you need to be aware of this when trying to measure something to a millisecond precision.

I’ve got something more interesting, a hardware input latency tester! It consists of a light sensor attached to a Teensy board, which in turn is plugged into the computer via USB.

I should really get around to writing a full blog post about this latency tester, but for now, you should read this post by Tristan Hume about building a similar device.¹ I used that post as a reference for building mine, but I wrote my own firmware and analysis scripts (these I am not sharing until they are less of an utter mess).

The main benefit of such a device is that it allows you to measure a full end-to-end input latency, including processing time in the kernel, the compositor, the application, and then the response time of the monitor itself. You are measuring what you really see and feel, excluding only the keyboard firmware (since the latency tester sends key press events directly over USB). There’s also very little extra load on the system, especially compared to using something like a screen capture API.

Here’s a gist of how it works. The light sensor is aimed at a specific, small area on the monitor, which will be affected by the key press (in our case, a specific character cell in the terminal). The board sends a key press over USB (for example, Space) and starts monitoring the light sensor readings. As soon as it detects a jump in the light amount, it releases the key. Then, it presses a second key (for example, Backspace) and waits for the light to change back. Now we’re back to square one; the firmware waits a randomized amount (to prevent “snapping” to the monitor refresh rate) and repeats the experiment.

During all of this process, the board dumps light sensor readings over a serial port as fast as it can manage (I’m getting about 35,500 readings per second with my current board and firmware). On the computer, I save all of this data into a file for offline analysis with Python code. This analysis code finds the timestamp where the light starts to change, and subtracts it from the timestamp of the key press, to get one input latency measurement.

I then aggregate the measurements and plot them with seaborn. Here’s an example of what the result looks like:

Input Latency Plots #

Let’s explore what you can find on this latency plot.

The small black dots represent the individual measurements. As in, every dot shows a real amount of time that had passed between one key press and the corresponding change in light on the sensor. There are 120 of these dots since I repeat each test 120 times.

Looking at the dots can confirm that the data is sensible. We expect the bulk of the measurements to be spread uniformly across an interval roughly the size of one monitor repaint cycle. This is because monitors generally repaint at a constant rate, and pressing a key at a random point in time should land us in a random point of the repaint cycle. We get the lowest latency if the application renders a new frame in response right in time for the monitor to show it. And we get the highest latency when the application finishes rendering a new frame just missing the monitor deadline, having to wait one extra repaint cycle for the pixel colors to change.

In the example above, the dots are spread over 7–8 ms, which is about equal to the ~6.94 ms refresh cycle of my 144 Hz monitor.

High outliers in the dots, or a larger spread, indicate lag or slowness of the application under test: some key presses are taking longer than others to process.

We do not expect to see any gaps between dot clusters. They would usually indicate aliasing with the monitor repaint cycle, or some frame scheduling bug in the compositor.²

The box shows statistics over the individual measurements:

median (a measurement perfectly “in the middle” with half of the measurements lower and half of the measurements higher),
lowest and highest measurement,
25th and 75th percentiles (with 25% and 75% of the measurements lower than the line, respectively).

All in all, you can compare applications by their spread, then by the median latency, and also look if there are any outliers.

With all that said, we’re almost ready to look at some results. I just need to tell you what exactly I was measuring the latency of.

Test Setup #

I did all tests on this system:

Lenovo Legion 7 Gen 7 AMD with Ryzen 7 6800H CPU and Radeon RX 6700M dGPU (using the dGPU exclusively via the MUX switch).
Monitor: Acer Nitro XV320QU, 2560×1440, 144 Hz, using 100% scale.
Host: Fedora 40 Silverblue Beta, Mesa 24.0.4.
Compositor: raw Mutter 46.0.

What is raw Mutter, you may ask? Well, Mutter is the compositor that GNOME Shell builds on top of. Turns out, you can start Mutter on its own, without GNOME Shell, by switching to a different VT and running a command like mutter --display-server -- alacritty. This gives you a very bare-bones environment that is only really meant for testing. It is, however, quite useful for benchmarking, as it represents something close to a zero-overhead GNOME Shell ideal case.

I’m testing several terminal applications. In the order of appearance on the plots, they are:

Alacritty: not VTE-based; serves as a baseline of sorts, because it is consistently one of the fastest terminals according to all of my prior tests.
Console: GTK 4, the default terminal in GNOME.³
VTE Test App: GTK 4, a test terminal that lives in the VTE repository.
GNOME Terminal: GTK 3,⁴ used to be the default in GNOME, and is still shipped out of the box in several distributions.

Since the intention is to compare GNOME 45 to GNOME 46, I used toolb\0x containers with Fedora 39 and Fedora 40 to install and run all terminals above, as packaged by Fedora with no extra tweaks.

I ran the terminals one by one and put their windows in the top left corner of the monitor. The mouse cursor was outside the window for all tests.⁵

Input Latency Tests #

The first test is simple: I run cat > /dev/null to get an input field with no readline or similar processing, and then I measure how long it takes for the terminal to move its block cursor one cell to the right after pressing Space.

This is meant to test the best possible scenario for the terminal, with the least overhead.

This is what the test process looks like:

And here are the results:

Alacritty, which is our baseline, did not change from F39 to F40, as expected.

But look at the massive improvement on all of the VTE terminals! They went from quite bad to pretty much on par with Alacritty, even the GTK 3 GNOME Terminal is very close.

The main change that caused this much improvement is likely this one by Christian that moves away from a 40 Hz VTE repaint timer to drawing every frame, synchronized with the monitor, as any self-respecting GTK widget should do.

Console has a few outliers which are maybe caused by its process tracking, but those are nothing new (they may be looked into for GNOME 47).

For the next test, I constructed a more realistic case. I took a snapshot of my neovim setup and opened the README from Ptyxis. I then strategically replaced a square of text with Unicode full-block characters to provide a bright “landing pad” for the light sensor.

The test consists of repeatedly pressing Ctrl+D and Ctrl+U to scroll the text buffer down and up in neovim. The light sensor alternates between an empty line (dark) and the full-block landing pad (bright). The neovim setup has a bunch of bells and whistles, so the terminal gets to have fun drawing the various underlines, undercurls, gutter icons, and the statusline.

This is what the test process looks like:

Here are the results:

The massive improvement is clear on this test too, and our GNOME 46 terminals are still pretty much on par with Alacritty!

Finally, let’s take a closer look at all Fedora 40 results on one plot:

This plot shows how much of a latency toll the neovim test takes compared to a simple cat, but the latency increase is similar across all terminals.

vtebench #

I also ran Alacritty’s vtebench suite across the same set of applications and configurations. This is a fully automated benchmark that measures something completely different from input latency: PTY read and parsing performance. ~~It has also proven quite capable at finding crashes in VTE.~~

Here’s what vtebench’s README has to say:

This benchmark is not sufficient to get a general understanding of the performance of a terminal emulator. It lacks support for critical factors like frame rate or latency. The only factor this benchmark stresses is the speed at which a terminal reads from the PTY. If you do not understand what this means, please do not jump to any conclusions from the results of this benchmark.

The repaint duration can and does affect the results of this test, especially for terminals that read and parse PTY on the same thread as they run their repaint logic, like VTE.

This is what one of the vtebench benchmarks looks like:

And here are the results:

To avoid making this plot even busier, I drew the green arrows on only one of the benchmarks. As you can see, other benchmarks show a similar trend.

VTE from GNOME 46 shows some welcome improvements here too, although a lot more varied, and not quite on par with Alacritty (which renders in a separate thread from reading and parsing). These improvements likely come from the many other optimizations that happened in VTE during the GNOME 46 cycle.

Note that I omitted two benchmarks from these results: dense_cells and unicode. They are the main stress tests of vtebench that hit the terminal really hard. Unfortunately, VTE still struggles with them and shows a huge spread, which pushes the rest of the results down and makes the plot less readable.

Open this to see the full results if you’re curious.

Conclusion #

VTE had a round of massive performance improvements in GNOME 46 which manifest as something you can really feel during normal terminal use. The input latency is down to almost matching the fastest terminals, even in a non-trivial neovim setup with lots of complexity on screen.

The remaining difference, at least on these test cases, is close to negligible. Some of it can be explained by VTE doing a bit more extra work for accessibility (enabled in GNOME Terminal and currently disabled in the GTK 4 terminals), scrollbar calculations, and other features.

If you’ve been avoiding VTE-based terminals due to sluggishness and input lag, now is the time to give them another chance. Just make sure you’re running VTE 0.76, which includes all of this goodness.

Huge thanks to the VTE maintainers and contributors for making this a reality, and congratulations on an awesome release!

P.S. If you’re curious about Ptyxis or the behavior of GTK’s NGL vs. NVK vs. GL renderers, they all perform similarly to the F40 VTE Test App results shown above. I did more extensive benchmarks of these a month ago, you can find them here.

As you can tell from the photo, I did not follow Tristan’s advice to make something fancier than just dangling wires. ↩︎
Just a few weeks ago some measurements I took showed a suspicious one-frame-long gap in the dots. And guess what, it was a frame scheduling bug in my compositor, with none other than myself to blame for it. Thankfully, it wasn’t hard to fix, and easy to verify afterward by redoing the same test. ↩︎
Your distribution may have a different idea of which terminal should be the default in its GNOME spin. For example, Fedora still ships GNOME Terminal by default. ↩︎
GNOME Terminal is being ported to GTK 4 for GNOME 47, but in GNOME 46 it is still a GTK 3 application. ↩︎
To avoid the link-under-cursor detection logic skewing the results. ↩︎

Identity v0.5 and Synchronized Scrolled Windows

Wed, 05 Apr 2023 12:46:58 -0700

My university studies and work revolve around image- and video-processing algorithms. I frequently need to compare similar but subtly different output videos: to see how various algorithms solving the same problem behave, or to see the progress as I’m tweaking my own algorithm.

In 2020, I made a GNOME app, Identity, to assist me. It plays multiple videos at once in sync, and lets you switch between them like tabs in a browser. This way you can easily examine the differences at any point.

Identity has seen a number of releases since then and grown a number of helpful features, like zooming or viewing media properties. And now, in v0.5, I have implemented a side-by-side comparison mode. All files are arranged in a row or a column, and their zoom and pan positions are synchronized. You can explore different parts of an image or a video and see how they look across all versions that you opened. This is a quite useful comparison mode, and also more obvious for first-time users.

Identity comparing an image with three upscaling methods in a column

Under the hood, every image sits inside a GtkScrolledWindow, the standard GTK 4 widget that provides scrolling/panning gestures for its child widget, and draws the scroll bars and the overshoot effect.¹ It’s easy to synchronize two or more of these scrolled windows together, but avoiding weird gesture interactions can be tricky. Let’s see how to get them to play along.

Synchronizing Positions #

Scrolled windows use GtkAdjustments to monitor the full size of their child widget and to control its current scroll position. Adjustments are objects with properties for the lower and upper bounds (in our case set to 0 and the full child size), the current value (which is the scroll position), and the step and page increments (which set how far arrow keys and PgUp/PgDown scroll the widget). There are two adjustments in every scrolled window: one for horizontal scrolling, and one for vertical scrolling, called hadjustment and vadjustment.

To synchronize multiple scrolled windows which show widgets of matching size, simply use the same two adjustments for all of them. Scrolling one widget will update the adjustments, causing all other widgets to also update their scroll position.

const shared_hadj = new Gtk.Adjustment();
const shared_vadj = new Gtk.Adjustment();

const scroll1 = new Gtk.ScrolledWindow({
    child: pictures[0],
    hadjustment: shared_hadj,
    vadjustment: shared_vadj,
});
const scroll2 = new Gtk.ScrolledWindow({
    // This pictures[1] widget has the same size as pictures[0].
    child: pictures[1],
    // Same adjustments as above!
    hadjustment: shared_hadj,
    vadjustment: shared_vadj,
});

You can run the full example with gjs -m simple.js:

Despite being a relatively simple and supported use-case, adjustment sharing actually makes conditions more favorable for an allocation loss bug that had plagued some of the more complex GTK 4 apps like Sysprof or GNOME Builder. When I implemented the initial version of side-by-side comparison in Identity, I started hitting the bug as well, very easily (panning a video while it was finishing and seeking back to the start was usually enough). So, I decided to investigate, and a few hours of rr and intense discussion in #gtk later, I managed to fix it! Of course, allocation machinery being very complex, this broke some things, but after a few follow-up fixes by the GTK maintainers, the bug seems to have been at last completely conquered. The fixes are included in GTK 4.10 and should make their way into GTK 4.8.4.

Anyhow, Identity can show and synchronize images of different size. Reusing the same adjustments would cause the upper boundaries to mismatch, and things to break. Instead, I keep track of my own, normalized adjustments, which always range from 0 to 1.² They are bound back and forth with the scrolled window adjustments, so that scrolling will cause an update to the normalized adjustments, and vice versa. In turn, the value of the normalized adjustments are bound together between all open images. This way, zooming into the center of one image will set the values to 0.5, which will scroll all other images into their centers, regardless of their current size.³

Finally, watch out for widgets which can change their size depending on the scroll position, like GtkListView with variably-sized items. Scrolling to a particular point may cause such a widget to update the upper boundary of the adjustment and recompute the scroll position relative to what it now believes to be its size. This may cause a cascading reaction with the synchronized widgets, and potentially an infinite loop.

Fixing Kinetic Scrolling #

Scrolled window implements kinetic deceleration for two-finger panning on a touchpad and one-finger panning on a touchscreen—if you swipe your fingers with some speed, the widget will keep scrolling for a bit, until it comes to a halt. At first it may seem that it works fine—you can try it in the simple example above—until you try to pan one widget, and then quickly pan the other widget, while the first one is still decelerating:

For this demonstration, I used the “Simulate Touchscreen” toggle in the Inspector

Something weird is happening: it’s like the widget doesn’t let you pan until the deceleration is over. The reason for this issue is that the pan gesture and the kinetic deceleration live in each scrolled window separately. So when you pan one scrolled window, it starts updating the (shared) adjustment value every frame, and if you try to pan another scrolled window in the meantime, the movement gets continuously overwritten by the first scrolled window.

The workaround is to stop kinetic deceleration on all other scrolled windows when starting the pan. It’s further complicated by the fact that the pan gestures themselves live inside the scrolled window, and you can’t mess with them. Thankfully, you can catch the two-finger touchpad gesture with a GtkEventControllerScroll and the one-finger touchscreen gesture with a GtkGestureDrag:

// Our scrolled windows, for stopping their kinetic scrolling.
const scrolledWindows = [];

function stopKineticScrollingExcluding(source) {
    for (const widget of scrolledWindows) {
        if (widget === source)
            continue;

        // There's no special function to stop kinetic scrolling,
        // but disabling and enabling it works fine.
        widget.set_kinetic_scrolling(false);
        widget.set_kinetic_scrolling(true);

        // Fix horizontal touchpad panning after resetting
        // kinetic scrolling.
        widget.queue_allocate();
    }
}

const shared_hadj = new Gtk.Adjustment();
const shared_vadj = new Gtk.Adjustment();

function createScrolledWindow() {
    // The scrollable widget.
    const picture = new Gtk.Picture({
        file: image,
        can_shrink: false,
    });

    const scrolledWindow = new Gtk.ScrolledWindow({
        child: picture,
        hadjustment: shared_hadj,
        vadjustment: shared_vadj,
    });
    scrolledWindows.push(scrolledWindow);

    // The scroll controller will catch touchpad pans.
    const scrollController = Gtk.EventControllerScroll.new(
        Gtk.EventControllerScrollFlags.BOTH_AXES,
    );
    scrollController.connect('scroll', (scrollController, _dx, _dy) => {
        const device = scrollController.get_current_event_device();
        if (device?.source === Gdk.InputSource.TOUCHPAD) {
            // A touchpad pan is about to start!
            // Let's stop the kinetic scrolling on other widgets.
            stopKineticScrollingExcluding(scrolledWindow);
        }

        // Let the default scrolling work.
        return false;
    });
    picture.add_controller(scrollController);

    // The drag gesture will catch touchscreen pans.
    const dragGesture = new Gtk.GestureDrag();
    dragGesture.connect('drag-begin', (dragGesture, _x, _y) => {
        const device = dragGesture.get_current_event_device();
        if (device?.source === Gdk.InputSource.TOUCHSCREEN) {
            // A touchscreen pan is about to start!
            // Let's stop the kinetic scrolling on other widgets.
            stopKineticScrollingExcluding(scrolledWindow);
        }

        // We don't want to handle the drag.
        dragGesture.set_state(Gtk.EventSequenceState.DENIED);
    });
    picture.add_controller(dragGesture);

    return scrolledWindow;
}

This gives us panning across all widgets with nice kinetic deceleration which doesn’t break. Try the full example with gjs -m reset-kinetic.js:⁴

Touchpad panning works as expected across all scrolled windows

There are two extra complications about this code, both related to touchscreen panning. First, we stop the kinetic scrolling on all scrolled windows excluding the one handling the new event. This is because for some reason resetting the kinetic scrolling like this in the middle of a touchscreen pan prevents it from working (touchpad pans keep working fine).

Second, we queue an allocation on the scrolled windows right after resetting the kinetic scrolling. For whatever reason, resetting the kinetic scrolling causes the scrolled window to stop handling horizontal touchscreen pans altogether (vertical and mixed pans keep working fine). I suspect it’s caused by some logic error related to check_attach_pan_gesture(). This function is called when toggling the kinetic scrolling, breaking the horizontal touchscreen pans. Thankfully, it’s also called at the end of allocation, where it fixes back the touchscreen pans. I haven’t investigated this bug further, but it would be nice to get it fixed.

And that’s it! The code we’ve added also comes in useful for implementing custom gestures like zoom or mouse pan. Just remember that when writing custom gestures, you might need to stop the kinetic scrolling on the current scrolled window too, not only on the linked ones.

Closing Thoughts #

When synchronizing scrolled windows, and just dealing with GTK gesture code in general, make sure to test with different input devices, as each has its own quirks. Be careful when scrollable widgets have different sizes, or can change their size depending on the scroll position.

At a higher level, I think it would be better if the kinetic deceleration lived somewhere around the GtkAdjustments themselves. This way it would also be shared between all synchronized scrolled windows, and the workarounds, along with their oddities, wouldn’t be necessary. Something to keep in mind for GTK 5 perhaps.

When discussing a draft of this post with GTK developers and contributors, another potential GTK 5 idea came up. Different scrollable widgets (GtkViewport, GtkListView, GtkTextView, WebKitGTK’s web view, libshumate’s map) have slightly different needs, and GtkScrollable with GtkScrolledWindow can’t offer them all a unified interface that would work without compromises or big technical hurdles. (The last two examples don’t implement the scrollable interface for these reasons.) So, maybe, instead of GtkScrolledWindow, there should be a collection of helpers, and scrollable widgets should show scrollbars and handle scrolling themselves.

With all that said, if you think Identity might be useful to you, download it from Flathub and give it a try! I’d love to hear your thoughts, ways to contact me are linked at the bottom of this page.

Comparing three videos side-by-side in Identity

The subtle glow that shows up when you try to scroll past the end of a scrollable widget. ↩︎
These normalized adjustments are also responsible for the behavior when resizing the Identity window with zoomed-in images: instead of always expanding to the bottom-left, the images expand around their current scroll position. This is because the normalized adjustments don’t change during resizing. So, for example, a value of 0.25 before and after resizing will keep the image scrolled to 25% of its size. ↩︎
This is not the only way to share position between differently sized scrollable widgets, just one that makes sense for Identity’s comparison use-case. You could imagine some other use-case where it makes more sense to share the pixel position, rather than the normalized position. It can be implemented using the same idea of two extra adjustments. It’ll work fine as long as different widgets don’t try to overwrite the upper bound on the same adjustment with different values. ↩︎
Unfortunately, “Simulate Touchscreen” won’t help you see this fix; you’ll need a real touchpad or touchscreen. At the moment, the toggle does not change the device types that the gesture code receives, so it doesn’t run our workaround code. To test Identity, I’ve been using the work-in-progress Mutter SDK branch which has a compositor-level touchscreen emulation. ↩︎

GSoC 2021: Overview

Sat, 21 Aug 2021 11:14:09 +0000

Over the summer I worked on implementing the new screenshot UI for GNOME Shell as part of Google Summer of Code 2021. This post is an overview of the work I did and work still left to do.

The project was about adding a dedicated UI to GNOME Shell for taking screenshots and recording screencasts. The idea was to unify related functionality in a discoverable and easy to use interface, while also improving on several aspects of existing screenshot and screencast tools.

Over the summer, I implemented most of the functionality:

Capturing screen and window snapshots immediately, letting the user choose what to save later.
Area selection, which can be resized and dragged after the first selection.
Screen selection.
Window selection presenting an Overview-like view.
Mouse cursor capturing which can be toggled on and off inside the UI.
Area and screen video recording.
Correct handling of HiDPI and mixed DPI setups.

I opened several merge requests:

The main GNOME Shell merge request with the screenshot UI.
A Mutter merge request adding a function to snapshot the screen into a GPU texture.
A Mutter merge request adding a function to get the scale of the cursor texture, required for correct mixed DPI handling.

I expect that Mutter merge requests won’t require many further changes before merging. The screenshot UI however still has some work that I will do past GSoC, detailed in the main merge request. This work includes adding window selection support for screen recording, ensuring all functionality is keyboard- and touch-accessible, and working with the designers to polish the final result. GNOME 41 is already past the UI freeze, but GNOME 42 seems to me like a realistic target for finishing and landing the screenshot UI.

For the purposes of GSoC, I additionally made two frozen snapshots of work done over the GSoC period that I will not update further: three commits in this mutter tag and 16 commits in this gnome-shell tag.

I also wrote several blog posts about my work on the screenshot UI:

GSoC 2021: GNOME Shell Screenshot UI, an introduction post showing the panel and the initial implementation of area and screen selection.
GSoC 2021: Selection Editing and Window Selection, a post showcasing handles for resizing the area selection, a better animation for opening the UI and window selection implementation.
GSoC 2021: Screenshots with Pointer, a post that details how I implemented showing mouse cursor on the screenshots and explains the challenges arising from mixed DPI.

Additionally, I gave a short presentation of my work at GUADEC, GNOME’s annual conference.

Over the course of this GSoC project I learned a lot about GNOME Shell’s UI internals which will help me with GNOME Shell contributions in the future. I enjoyed working on an awesome upgrade to taking screenshots and screencasts in GNOME. For me participating in the GNOME community is a fantastic experience and I highly recommend everyone to come hang out and contribute.

I would like to once again thank my mentor Jonas Dreßler for answering my questions, as well as Tobias Bernard, Allan Day and Jakub Steiner for providing design feedback.

Screen recording in the new screenshot UI

GSoC 2021: Screenshots with Pointer

Fri, 06 Aug 2021 17:40:37 +0000

Over the summer I’m working on a new screenshot UI for GNOME Shell. Here’s my progress since the last post.

The new “Show Pointer” toggle in the screenshot UI

First of all, I made the window selection mode work across multiple screens and ensured that it works correctly with HiDPI and mixed DPI setups. Each screen gets its own Overview-like view of all the windows, letting you pick the one you need at your leisure.

In this and the following showcases, you can see GNOME Shell running with two virtual monitors: one regular DPI on the left, and one high DPI (200% scaling) on the right. Both virtual monitors use the same resolution, which is why the right one appears two times smaller.

Window selection working across two monitors

Next, I implemented the screen selection mode which lets you choose a full monitor to screenshot.

Screen selection with the primary monitor selected

Finally, I embarked on an adventure to add a “Show Pointer” toggle. Following the spirit of the screenshot UI, you should be able to hit your Print Screen key first and adjust the screenshot contents afterwards. That is, you should be able to show and hide the mouse pointer and see it on the preview in real-time.

But first things first: let’s figure out how to add a menu. There’s a handy PopupMenu class that you can inherit to make your own menu:

class UIMenu extends PopupMenu.PopupMenu {
    constructor(sourceActor) {
        // The third argument controls which side
        // the menu "points" to. Here the menu
        // will point to the left.
        super(sourceActor, 0, St.Side.LEFT);

        Main.uiGroup.add_actor(this.actor);
        this.actor.hide();
    }

    toggle() {
        if (this.isOpen)
            this.close(BoxPointer.PopupAnimation.FULL);
        else
            this.open(BoxPointer.PopupAnimation.FULL);
    }
}

To show the menu on a button press, we also need a PopupMenuManager:

let button = new St.Button();

let menu = new UIMenu(button);
let manager = new PopupMenu.PopupMenuManager(button);
manager.addMenu(menu);

button.connect('clicked', () => menu.toggle());

Let’s add a switch to our menu. PopupSwitchMenuItem is exactly what we need:

class UIMenu extends PopupMenu.PopupMenu {
    constructor(sourceActor) {
        // ...

        this._showPointerItem =
            new PopupMenu.PopupSwitchMenuItem(_("Show Pointer"), false);
        this._showPointerItem.connect(
            'toggled', (_item, state) => {
                this.emit('show-pointer-toggled', state);
            });
        this.addMenuItem(this._showPointerItem);
    }

    get showPointer() {
        return this._showPointerItem.state;
    }

    // ...
}
Signals.addSignalMethods(UIMenu.prototype);

Pay attention to the last line. Signals.addSignalMethods() does a bit of magic that lets you use GObject signal methods (connect() and emit()) on plain JavaScript classes. In this case I use it to thread through a signal for toggling the “Show Pointer” switch.

The mouse cursor on the preview is just another St widget. Its visibility is connected to the state of the “Show Pointer” switch:

let cursor = new St.Widget();

menu.connect('show-pointer-toggled', (_menu, state) => {
    cursor.visible = state;
});

// Set the initial state.
cursor.visible = menu.showPointer;

When screenshot UI captures a snapshot of the screen, it will also snapshot the current cursor texture, position and scale. These variables are used to configure the cursor widget so it shows in the same spot in the screenshot UI as where it was on screen:

// Get a snapshot of the screen contents.
let [content, scale, cursorContent, cursorPoint, cursorScale] =
    await screenshot.to_content();

// Set the cursor texture.
cursor.set_content(cursorContent);
// Set the cursor position.
cursor.set_position(cursorPoint.x, cursorPoint.y);

// Get the cursor texture size.
let [, w, h] = cursorContent.get_preferred_size();

// Adjust it according to the cursor scale.
w *= cursorScale;
h *= cursorScale;

// Set the cursor size.
cursor.set_size(w, h);

The scale is needed mainly for HiDPI setups. Clutter operates in logical pixels, which means that, for example, on a monitor with 200% scaling, a widget with a size of 10×10 will occupy a 20×20 physical pixel area. Since get_preferred_size() returns a size in physical pixels, we need to multiply it by cursorScale to convert it to logical pixels.

With this, we have a working cursor preview in the screenshot UI:

How many layers of screenshot UI were used to take this picture?

When writing the final screenshot, we need to composite the cursor texture on the screenshot image. To do it correctly, we need to take into account scale of the screenshot texture, scale of the cursor texture, screen selection and cursor coordinates:

Shell.Screenshot.capture_from_texture(
    // The screen texture.
    texture,
    // Selected area.
    x, y, w, h,
    // Scale of the screen texture.
    scale,
    // The cursor texture.
    cursorTexture,
    // Cursor coordinates in physical pixels.
    cursor.x * scale,
    cursor.y * scale,
    // Scale of the cursor texture.
    cursorScale,
    // ...
);

With this in place, cursor capturing works perfectly across mixed screen and cursor texture scales:

Previewing and capturing the cursor in various configurations

But we’re not done yet! Time for window selection.

In window selection mode, every window gets its own cursor preview sprite since the cursor can overlap multiple windows at once:

Overlapping cursor in screen selection and window selection modes

If you thought scale handling was complicated above, brace yourself because window selection takes it a level further. Apart from the scale of the window buffer (counter-part to the screenshot texture scale from before) and the scale of the cursor texture, there’s also the scale that overview-like window selection applies to windows to fit them all on screen. To handle all of this complex positioning, I overrode the allocate() virtual function of the window preview actor:

vfunc_allocate(box) {
    this.set_allocation(box);

    // Window buffer size in physical pixels.
    let [, windowW, windowH] =
        this.content.get_preferred_size();

    // Compute window scale.
    //
    // Divide by buffer scale to convert
    // from physical to logical pixels.
    let xScale =
        (box.x2 - box.x1) /
        (windowW / this._bufferScale);
    let yScale =
        (box.y2 - box.y1) /
        (windowH / this._bufferScale);

    let cursor = this.get_child();

    // Compute cursor size in logical pixels.
    let [, , w, h] =
        cursor.get_preferred_size();
    w *= this._cursorScale;
    h *= this._cursorScale;

    // The cursor position and size.
    let cursorBox = new Clutter.ActorBox({
        x1: this._cursorPoint.x,
        y1: this._cursorPoint.y,
        x2: this._cursorPoint.x + w,
        y2: this._cursorPoint.y + h,
    });

    // Rescale it to match the window scale.
    cursorBox.x1 *= xScale;
    cursorBox.x2 *= xScale;
    cursorBox.y1 *= yScale;
    cursorBox.y2 *= yScale;

    // Allocate the cursor.
    cursor.allocate(cursorBox);
}

Finally, we need to pass these values to the recording function in a similar fashion to what we did before:

Shell.Screenshot.capture_from_texture(
    // The window texture.
    texture,
    // Special values that mean
    // "record the whole texture".
    0, 0, -1, -1,
    // Scale of the window texture.
    window.bufferScale,
    // The cursor texture.
    cursorTexture,
    // Cursor coordinates in physical pixels.
    window.cursorPoint.x * window.bufferScale,
    window.cursorPoint.y * window.bufferScale,
    // Scale of the cursor texture.
    cursorScale,
    // ...
);

Phew! Now we can lean back and enjoy window screenshots with cursor working perfectly across various screen, window and cursor scales. Don’t forget the cursor can be toggled on and off after the fact—this is what all the trouble was for!

Cursor capture on window selection

With pointer capturing implemented (although with some minor bugfixes still due), the next step is screen recording. You should be able to select an area, a monitor, or a window to record, optionally with a cursor, and start the recording. The design for what happens next is not finalized yet but a natural place to put the recording indicator and the stop button seems to be the top-right menu on the panel.

Thanks for getting all the way through the post and see you in the next update! By the way, check out my GUADEC intern lightning talk about the new screenshot UI in this YouTube recording.

GSoC 2021: Selection Editing and Window Selection

Thu, 15 Jul 2021 12:39:43 +0000

This summer I’m implementing a new screenshot UI for GNOME Shell. In this post I’ll show my progress over the past two weeks.

The new screenshot UI in the area selection mode

I spent the most time adding the four corner handles that allow you to adjust the selection. GNOME Shell’s drag-and-drop classes were mostly sufficient, save for a few minor things. In particular, I ended up extending the _Draggable class with a drag-motion signal emitted every time the dragged actor’s position changes. I used this signal to update the selection rectangle coordinates so it responds to dragging in real-time without any lag, just as one would expect. Some careful handling was also required to allow dragging the handle past selection edges, so for example it’s possible to grab the top-left handle and move it to the right and to the bottom, making it a bottom-right handle.

Editing the selection by dragging the corner handles

I’ve also implemented a nicer animation when opening the screenshot UI. Now the screen instantly freezes when you press the Print Screen button and the screenshot UI fades in, without the awkward screenshot blend. Here’s a side-by-side comparison to the previous behavior:

Comparison of the old and new opening animation, slowed down 2×

Additionally, I fixed X11 support for the new screenshot capturing. Whereas on Wayland the contents of the screen are readily available because GNOME Shell is responsible for all screen compositing, on X11 that’s not always the case: full-screen windows get unredirected, which means they bypass the compositing and go straight through the X server to the monitor. To capture a screenshot, then, GNOME Shell first needs to disable unredirection for one frame and paint the stage.

This X11 capturing works just as well as on Wayland, including the ability to capture transient windows such as tooltips—a long-requested feature. However, certain right-click menus on X11 grab the input and prevent the screenshot UI hotkey (and other hotkeys such as Super to enter the Overview) from working. This has been a long-standing limitation of the X11 session; unfortunately, these menus cannot be captured on X11. On Wayland this is not a problem as GNOME Shell handles all input itself, so windows cannot block its hotkeys.

Finally, over the past few days I’ve been working on window selection. Similarly to full-screen screenshots, every window’s contents are captured immediately as you open the screenshot UI, allowing you to pick the right window at your own pace. To capture the window contents I use Robert Mader’s implementation, which I invoke for all windows from the current workspace when the screenshot UI is opening. I arrange these window snapshots in a grid similar to the Overview and let the user pick the right window.

Window selection in action

As usual, the design is nowhere near finished or designer-approved. Consider it an instance of my “programmer art”. 😁

My goal was to re-use as much of the Overview window layout code as possible. I ended up making my own copy of the WorkspaceLayout class (I was able to strip it down considerably because the original class has to deal with windows disappearing, re-appearing and changing size, whereas the screenshot UI window snapshots never change) and directly re-using the rest of the machinery. I also made my own widget compatible with WindowPreview, which exports the few functions used by the layout code, once again considerably simplified thanks to not having to deal with the ever changing real windows.

The next step is to put more work into the window selection to make sure it handles all the different setups and edge cases right: the current implementation is essentially the first working draft that only supports the primary monitor. Then I’ll need to add the ability to pick the monitor in the screen selection mode and make sure it works fine with different setups too. I also want to figure out capturing screenshots with a visible cursor, which is currently notably missing from the screenshot UI. After that I’ll tackle the screen recording half.

Also, unrelated to the screenshot UI, I’m happy to announce that my merge request for reducing input latency in Mutter has finally been merged and should be included in Mutter 41.alpha.

That’s it for this post, see you in the next update!

GSoC 2021: GNOME Shell Screenshot UI

Tue, 29 Jun 2021 12:36:21 +0000

Hello! I’m Ivan Molodetskikh, a computer science student from Moscow, Russia.

I’ve been involved in GNOME starting from my GSoC 2018 project to port librsvg filters to Rust. Throughout the last year in GNOME I’ve been doing some work to reduce input latency in Mutter, the GNOME’s compositor (by implementing the presentation-time Wayland protocol and adding dynamic render time computation). I’ve also created two small apps, Video Trimmer and Identity.

As part of this year’s Google Summer of Code, I’m implementing a new screenshot UI in GNOME Shell.

Screenshot UI panel mock-up by the design team

The UI will make taking screenshots and recording screencasts more intuitive and discoverable. On a key press, GNOME Shell will capture a full screenshot, and you will be able to select the exact area you want. The screenshot is captured immediately, so it’s much easier to catch the right moment or capture open context menus.

Screencasts will get an upgrade too: you will be able to record areas of the screen or individual windows, just like you already can with screenshots.

Over the first few weeks I figured out how to add new UI elements to GNOME Shell: how to construct UI with GJS, how to style elements with CSS, the difference between Clutter actors and layouts and St (GNOME Shell’s toolkit) widgets, how to do transitions and handle input. I’ve been basing my work on the UI mock-up from the design team. Here’s a short demo of what I’ve implemented so far:

Demo of the parts of the mock-up that I’ve implemented thus far

Keep in mind this is very much a work-in-progress: I used stock icons instead of correct mock-up ones, I haven’t got any designer feedback yet, screen recording is not implemented and so on.

Using Robert Mader’s texture actor implementation, I added a Mutter function to snapshot the screen contents into a GPU texture that can be shown on a GNOME Shell widget. This way I can instantly display the screenshot preview in the UI without doing a slow PNG encoding round-trip. Then the UI allows you to select an area or a screen and record it into an image by pressing the capture button. Currently, the image is copied into the clipboard. I paste the screenshot into Obfuscate to display it.

When switching into the screencast mode, instead of the screen snapshot you can simply see your desktop normally because screen recording starts only upon pressing the capture button, not from an old screen snapshot.

The next step is to implement Window selection, which will arrange windows similarly to the Overview. Afterwards I’ll work on the screen recording part. I have also contacted the design team to get feedback and make sure the UI is the best it can be.

I’d like to thank my mentor, Jonas Dreßler (aka verdre), for keeping up with my questions. I’m excited to bring an awesome screenshot UI to GNOME, see you all in the next blog posts!

GSoC 2018: Overview

Fri, 10 Aug 2018 13:08:36 +0000

Introduction #

Throughout the summer I was working on librsvg, a GNOME library for rendering SVG files to Cairo surfaces. This post is an overview of the work I did with relevant links.

My Results #

For the project I was to port the SVG filter infrastructure of librsvg from C to Rust, adding all missing filter tests from the SVG test suite along the way. I was also expected to implement abstractions to make the filter implementation more convenient, including Rust iterators over the surface pixels. Here’s a list of all merge requests accepted into librsvg as part of my GSoC project:

Splitting the existing C filter code into individual files for each filter:
- merge request.
Porting of the filter infrastructure to Rust, fixing bugs, implementing missing features and adding tests:
Improving performance of filters:
Miscellaneous fixes:
- enabling debuginfo in release builds,
- distcheck fix.

Here’s a convenient link to see all of these merge requests in GitLab: https://gitlab.gnome.org/GNOME/librsvg/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=YaLTeR&label_name[]=GSoC%202018

All of this code was accepted into the mainline and will appear in the next stable release of librsvg. I also wrote the following blog posts detailing some interesting things I worked on as part of the GSoC project:

GSoC 2018: Filter Infrastructure talking about the filter infrastructure, pixel iterators and benchmarking and showing an example of implementing a filter primitive;
GSoC 2018: Safe Shared Access to Cairo Image Surfaces showcasing the abstractions for safe shared access to Cairo image surface pixel data and showing how it can enable additional optimizations on the example of the alpha-only optimizations;
GSoC 2018: Parallelizing Filters with Rayon covering the parallelization of two computationally-intensive filters using the rayon crate.

Further Work #

There are a couple of fixes which still need to be done for filters to be feature-complete:

Fixing filters operating on off-screen nodes. Currently all intermediate surfaces are limited to the original SVG view area so anything off-screen is inaccessible to filters even when it should be. This is blocked on some considerable refactoring in the main librsvg node drawing code which is currently underway.
Implementing the filterRes property. This property allows to set the pixel resolution for filter operations and is one of the ways of achieving more resolution-independent rendering results. While it can be implemented with the current code as is, it will be much more convenient to account for it while refactoring the code to fix the previous issue.
Implementing the enable-background property. The BackgroundImage filter input should adhere to this property when picking which nodes to include in the background image, whereas it currently doesn’t.

GSoC 2018: Parallelizing Filters with Rayon

Fri, 10 Aug 2018 12:08:10 +0000

Introduction #

I’m working on SVG filter effects in librsvg, a GNOME library for rendering SVG files to Cairo surfaces. After finishing porting all filters from C to Rust and adding tests, I started investigating the filter performance. With the codebase converted to Rust, I am able to confidently apply important optimizations such as parallelization. In this post I’ll show how I parallelized two computation-intensive filter primitives.

Rayon #

Rayon is a Rust crate for introducing parallelism into existing code. It utilizes Rust’s type system to guarantee memory safety and data-race free execution. Rayon mimics the standard Rust iterators, so it’s possible to convert the existing iterator-using code with next to no changes. Compare the following single-threaded and parallel code:

// Single-threaded.
fn sum_of_squares(input: &[i32]) -> i32 {
    input.iter()
         .map(|i| i * i)
         .sum()
}

// Parallelized with rayon.
use rayon::prelude::*;
fn sum_of_squares(input: &[i32]) -> i32 {
    input.par_iter()
         .map(|i| i * i)
         .sum()
}

By merely using .par_iter() instead of .iter(), the computation becomes parallelized.

Parallelizing Lighting Filters #

Going forward with analyzing and improving the performance of the infamous mobile phone case, the two biggest time sinks were the lighting and the Gaussian blur filter primitives. It can be easily seen on the callgrind graph from KCachegrind:

Since I was working on optimizing the lighting filters, I decided to try out parallelization there first. The lighting filter primitives in SVG (feDiffuseLighting and feSpecularLighting) can be used to cast light onto the canvas using a render of an existing SVG node as a bump map. The computation is quite involved, but it boils down to performing a number of arithmetic operations for each pixel of the input surface independently of the others—a perfect target for parallelization. This is what the code initially looked like:

let mut output_surface = ImageSurface::create(
    cairo::Format::ARgb32,
    input_surface.width(),
    input_surface.height(),
)?;

let output_stride = output_surface.get_stride() as usize;
let mut output_data = output_surface.get_data().unwrap();

let mut compute_output_pixel = |x, y, normal: Normal| {
    let output_pixel = /* expensive computations */;

    output_data.set_pixel(output_stride, output_pixel, x, y);
};

// Compute the edge pixels
// <...>

// Compute the interior pixels
for y in bounds.y0 as u32 + 1..bounds.y1 as u32 - 1 {
    for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
        compute_output_pixel(
            x,
            y,
            interior_normal(&input_surface, bounds, x, y),
        );
    }
}

The edge pixel computation is separated out for optimization reasons and it’s not important. We want to focus on the main loop over the interior pixels: it takes up the most time. What we’d like to do is to take the outer loop over the image rows and run it in parallel on a thread pool. Since each row (well, each pixel in this case) is computed independently of the others, we should be able to do it without much hassle. However, we cannot do it right away: the compute_output_pixel closure mutably borrows output_data, so sharing it over multiple threads would mean multiple threads get concurrent mutable access to all image pixels, which could result in a data race and so won’t pass the borrow checker of the Rust compiler. Instead, we can split the output_data slice into row-sized non-overlapping chunks and feed each thread only those chunks that it needs to process. This way no thread can access the data of another thread. Let’s change the closure to accept the target slice (as opposed to borrowing it from the enclosing scope). Since the slices will start from the beginning of each row rather than the beginning of the whole image, we’ll also add an additional base_y argument to correct the offsets.

let compute_output_pixel =
    |mut output_slice: &mut [u8],
     base_y,
     x,
     y,
     normal: Normal| {
        let output_pixel = /* expensive computations */;

        output_slice.set_pixel(
            output_stride,
            output_pixel,
            x,
            y - base_y,
        );
    };

// Compute the interior pixels
for y in bounds.y0 as u32 + 1..bounds.y1 as u32 - 1 {
    for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
        compute_output_pixel(
            output_data,
            0,
            x,
            y,
            interior_normal(&input_surface, bounds, x, y),
        );
    }
}

Now we can convert the outer loop to operate through iterators using the chunks_mut() method of a slice which does exactly what we want: returns the slice in evenly sized non-overlapping mutable chunks.

let compute_output_pixel =
    |mut output_slice: &mut [u8],
     base_y,
     x,
     y,
     normal: Normal| {
        let output_pixel = /* expensive computations */;

        output_slice.set_pixel(
            output_stride,
            output_pixel,
            x,
            y - base_y,
        );
    };

// Compute the interior pixels
let first_row = bounds.y0 as u32 + 1;
let one_past_last_row = bounds.y1 as u32 - 1;
let first_pixel = (first_row as usize) * output_stride;
let one_past_last_pixel =
    (one_past_last_row as usize) * output_stride;

output_data[first_pixel..one_past_last_pixel]
    .chunks_mut(output_stride)
    .zip(first_row..one_past_last_row)
    .for_each(|(slice, y)| {
        for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
            compute_output_pixel(
                slice,
                y,
                x,
                y,
                interior_normal(
                    &input_surface,
                    bounds,
                    x,
                    y,
                ),
            );
        }
    });

And finally, parallelize by simply changing chunks_mut() to par_chunks_mut():

use rayon::prelude::*;

output_data[first_pixel..one_past_last_pixel]
    .par_chunks_mut(output_stride)
    .zip(first_row..one_past_last_row)
    .for_each(|(slice, y)| {
        for x in bounds.x0 as u32 + 1..bounds.x1 as u32 - 1 {
            compute_output_pixel(
                slice,
                y,
                x,
                y,
                interior_normal(
                    &input_surface,
                    bounds,
                    x,
                    y,
                ),
            );
        }
    });

Let’s see if the parallelization worked! Here I’m using time to measure how long it takes to render the mobile phone SVG. Before parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
6.95user 0.66system **0:07.62**elapsed **99%**CPU (0avgtext+0avgdata 270904maxresident)k
0inputs+2432outputs (0major+714373minor)pagefaults 0swaps

After parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
7.47user 0.63system **0:06.04**elapsed **134%**CPU (0avgtext+0avgdata 271328maxresident)k
0inputs+2432outputs (0major+714460minor)pagefaults 0swaps

Note that even though the user time went up, the elapsed time went down by 1.5 seconds, and the CPU utilization increased past 100%. Success!

Parallelizing Gaussian Blur #

Next, I set out to parallelize Gaussian blur, the biggest timesink for the phone and arguably one of the most used SVG filters altogether. The SVG specification hints that for all reasonable values of the standard deviation parameter the blur can be implemented as three box blurs (taking the average value of the pixels) instead of the much more costly Gaussian kernel convolution. Pretty much every SVG rendering agent implements it this way since it’s much faster and librsvg is no exception. This is why you’ll see functions called box_blur and not gaussian_blur. Both box blur and Gaussian blur are separable convolutions, which means it’s possible to implement them as two passes, one of which is a loop blurring each row of the image and another is a loop blurring each column of the image independently of the others. For box blur specifically it allows for a much more optimized convolution implementation. In librsvg, the box blur function contains an outer loop over the rows or the columns of the input image, depending on the vertical argument and an inner loop over the columns or the rows, respectively. It uses i and j for the outer and inner loop indices and has some helper functions to convert those to the actual coordinates, depending on the direction.

// Helper functions for getting and setting the pixels.
let pixel = |i, j| {
    let (x, y) = if vertical { (i, j) } else { (j, i) };

    input_surface.get_pixel_or_transparent(bounds, x, y)
};

let mut set_pixel = |i, j, pixel| {
    let (x, y) = if vertical { (i, j) } else { (j, i) };

    output_data.set_pixel(output_stride, pixel, x, y);
};

// Main loop
for i in other_axis_min..other_axis_max {
    // Processing the first pixel
    // <...>

    // Inner loop
    for j in main_axis_min + 1..main_axis_max {
        // <...>
    }
}

Trying to convert this code to use chunks_mut() just like the lighting filters, we stumble on an issue: if the outer loop is iterating over columns, rather than rows, the output slices for all individual columns overlap (because the pixels are stored in row-major order). We need some abstraction, like a matrix slice, that can be split into non-overlapping mutable subslices by rows or by columns. The first thing that comes to mind is to try using the Matrix type from the nalgebra crate which does have that functionality. However, it turns out that nalgebra doesn’t currently support rayon or even have by-row or by-column iterators. I tried implementing my own iterators but that required some very non-obvious unsafe code with odd trait bound restrictions which I really wasn’t sure were correct. Thus, I scrapped that code and made my own wrapper for the ImageSurface which only contains things needed for this particular use case. To reiterate, we need a wrapper that:

provides write access to the image pixels,
can be split by row or by column into non-overlapping chunks,
is Send, i.e. can be safely sent between threads (for parallelizing).

Here’s what I came up with:

struct UnsafeSendPixelData<'a> {
    width: u32,
    height: u32,
    stride: isize,
    ptr: NonNull<u8>,
    _marker: PhantomData<&'a mut ()>,
}

unsafe impl<'a> Send for UnsafeSendPixelData<'a> {}

impl<'a> UnsafeSendPixelData<'a> {
    /// Creates a new `UnsafeSendPixelData`.
    ///
    /// # Safety
    /// You must call `cairo_surface_mark_dirty()` on the
    /// surface once all instances of `UnsafeSendPixelData`
    /// are dropped to make sure the pixel changes are
    /// committed to Cairo.
    #[inline]
    unsafe fn new(
        surface: &mut cairo::ImageSurface,
    ) -> Self {
        assert_eq!(
            surface.get_format(),
            cairo::Format::ARgb32
        );
        let ptr = surface.get_data().unwrap().as_mut_ptr();

        Self {
            width: surface.get_width() as u32,
            height: surface.get_height() as u32,
            stride: surface.get_stride() as isize,
            ptr: NonNull::new(ptr).unwrap(),
            _marker: PhantomData,
        }
    }

    /// Sets a pixel value at the given coordinates.
    #[inline]
    fn set_pixel(&mut self, pixel: Pixel, x: u32, y: u32) {
        assert!(x < self.width);
        assert!(y < self.height);

        let value = pixel.to_u32();

        unsafe {
            let ptr = self.ptr.as_ptr().offset(
                y as isize * self.stride + x as isize * 4,
            ) as *mut u32;
            *ptr = value;
        }
    }

    /// Splits this `UnsafeSendPixelData` into two at the
    /// given row.
    ///
    /// The first one contains rows `0..index` (index not
    /// included) and the second one contains rows
    /// `index..height`.
    #[inline]
    fn split_at_row(self, index: u32) -> (Self, Self) {
        assert!(index <= self.height);

        (
            UnsafeSendPixelData {
                width: self.width,
                height: index,
                stride: self.stride,
                ptr: self.ptr,
                _marker: PhantomData,
            },
            UnsafeSendPixelData {
                width: self.width,
                height: self.height - index,
                stride: self.stride,
                ptr: NonNull::new(unsafe {
                    self.ptr
                        .as_ptr()
                        .offset(index as isize * self.stride)
                }).unwrap(),
                _marker: PhantomData,
            },
        )
    }

    /// Splits this `UnsafeSendPixelData` into two at the
    /// given column.
    ///
    /// The first one contains columns `0..index` (index not
    /// included) and the second one contains columns
    /// `index..width`.
    #[inline]
    fn split_at_column(self, index: u32) -> (Self, Self) {
        assert!(index <= self.width);

        (
            UnsafeSendPixelData {
                width: index,
                height: self.height,
                stride: self.stride,
                ptr: self.ptr,
                _marker: PhantomData,
            },
            UnsafeSendPixelData {
                width: self.width - index,
                height: self.height,
                stride: self.stride,
                ptr: NonNull::new(unsafe {
                    self.ptr
                        .as_ptr()
                        .offset(index as isize * 4)
                }).unwrap(),
                _marker: PhantomData,
            },
        )
    }
}

The wrapper contains a pointer to the data rather than a mutable slice of the data, so the intermediate pixels (which cannot be accessed through set_pixel()) are not mutably aliased between different instances of UnsafeSendPixelData. Now it’s possible to implement an iterator over the rows or the columns for this wrapper, however I went with a different, simpler approach: using rayon’s scope functionality which allows spawning worker threads directly into rayon’s thread pool. First, let’s change the existing code to operate on individual rows or columns, just like we did with the lighting filters:

// The following loop assumes the first row or column of
// `output_data` is the first row or column inside `bounds`.
let mut output_data = if vertical {
    output_data.split_at_column(bounds.x0 as u32).1
} else {
    output_data.split_at_row(bounds.y0 as u32).1
};

for i in other_axis_min..other_axis_max {
    // Split off one row or column and launch its processing
    // on another thread. Thanks to the initial split before
    // the loop, there's no special case for the very first
    // split.
    let (mut current, remaining) = if vertical {
        output_data.split_at_column(1)
    } else {
        output_data.split_at_row(1)
    };

    output_data = remaining;

    // Helper function for setting the pixels.
    let mut set_pixel = |j, pixel| {
        // We're processing rows or columns one-by-one, so
        // the other coordinate is always 0.
        let (x, y) = if vertical { (0, j) } else { (j, 0) };
        current.set_pixel(pixel, x, y);
    };

    // Processing the first pixel
    // <...>

    // Inner loop
    for j in main_axis_min + 1..main_axis_max {
        // <...>
    }
}

I could avoid the current and the base_i arguments to the set_pixel closure because I can declare the closure from within the loop, whereas in the lighting filters code the compute_output_pixel closure had to be used and so declared outside of the main loop. Now it’s a simple change to split the work across rayon’s threads:

// The following loop assumes the first row or column of
// `output_data` is the first row or column inside `bounds`.
let mut output_data = if vertical {
    output_data.split_at_column(bounds.x0 as u32).1
} else {
    output_data.split_at_row(bounds.y0 as u32).1
};

// Establish a scope for the threads.
rayon::scope(|s| {
    for i in other_axis_min..other_axis_max {
        // Split off one row or column and launch its
        // processing on another thread. Thanks to the
        // initial split before the loop, there's no special
        // case for the very first split.
        let (mut current, remaining) = if vertical {
            output_data.split_at_column(1)
        } else {
            output_data.split_at_row(1)
        };

        output_data = remaining;

        // Spawn the thread for this row or column.
        s.spawn(move |_| {
            // Helper function for setting the pixels.
            let mut set_pixel = |j, pixel| {
                // We're processing rows or columns
                // one-by-one, so the other coordinate is
                // always 0.
                let (x, y) =
                    if vertical { (0, j) } else { (j, 0) };
                current.set_pixel(pixel, x, y);
            };

            // Processing the first pixel
            // <...>

            // Inner loop
            for j in main_axis_min + 1..main_axis_max {
                // <...>
            }
        });
    }
});

Let’s measure the performance. In the end of the previous section we had:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
7.47user 0.63system **0:06.04**elapsed **134%**CPU (0avgtext+0avgdata 271328maxresident)k
0inputs+2432outputs (0major+714460minor)pagefaults 0swaps

And now after the parallelization:

└─ time ./rsvg-convert -o temp.png mobile_phone_01.svg
10.32user 1.10system **0:04.57**elapsed **250%**CPU (0avgtext+0avgdata 272588maxresident)k
0inputs+2432outputs (0major+1009498minor)pagefaults 0swaps

We cut another 1.5 seconds and further increased the CPU utilization!

Conclusion #

Rayon is an excellent crate which provides a multitude of ways for safely parallelizing Rust code. It builds on idiomatic concepts such as iterators, but also contains a number of other convenient ways for parallelizing code when standard iterators aren’t sufficient or wouldn’t be very convenient. It uses the Rust’s type system to statically guarantee the absence of data races and manages the low-level details on its own allowing the programmer to focus on the actual computation. The parallelized filters are now included in the latest development release of librsvg for testing if using multiple threads leads to any issues downstream.

GSoC 2018: Safe Shared Access to Cairo Image Surfaces

Mon, 16 Jul 2018 09:53:00 +0000

Introduction #

I’m working on librsvg, a GNOME SVG rendering library, to port the SVG filter effects and related infrastructure from C to Rust. Librsvg uses Cairo, a 2D graphics library, for most of its drawing operations. Cairo can draw to a number of different surfaces like XCB and Xlib windows and pixmaps, PDF documents and PostScript files.

Image Surfaces #

Filter effects operate on rasterized bitmaps, so most of them need direct access to pixel data. There’s a special Cairo surface type for that: an image surface, backed by a memory buffer. Filters receive their input images in image surfaces, perform the necessary pixel operations and return new image surfaces with the results. In the Rust bindings, image surfaces are represented by the ImageSurface struct. It has a get_data() method which returns an ImageSurfaceData, which in turn acts as a slice into the underlying memory. Since ImageSurfaceData gives mutable access to the surface memory, it must ensure there are no other views into the same memory to comply with Rust’s ownership rules. This is achieved by checking that the ImageSurface reference count is equal to 1 in get_data(), which means that no other references to the ImageSurface exist. Furthermore, the ImageSurface is borrowed mutably for the lifetime of ImageSurfaceData. This is needed to prevent cloning the surface after obtaining ImageSurfaceData and subsequently drawing on it with a Cairo context while the view into the surface memory still exists. While this scheme does work, it offers only mutable, unique access to the pixel data. In the filter code, it’s much more convenient to have multiple references to the input surfaces with read-only memory access. Simply adding a struct similar to ImageSurfaceData which provides only a read-only view into the memory does not allow to drop the unique reference constraint, because it’s always possible to use the other reference to concurrently mutate the pixel data.

Shared Image Surface #

To work around the constraints, I ended up creating a special wrapper for ImageSurfaces:

#[derive(Debug, Clone)]
struct SharedImageSurface {
    surface: ImageSurface,

    data_ptr: NonNull<u8>, // *const.
    width: i32,
    height: i32,
    stride: isize,
}

The idea is to wrap a unique ImageSurface and provide just read-only access to it, while allowing cloning of SharedImageSurface itself. This way there can be multiple SharedImageSurfaces without compromising soundness because without direct access to the underlying ImageSurface it’s impossible to mutate it in any way from the outside. Additionally, since we know the surface won’t be modified, we can cache some common properties to get rid of extra C calls which can’t be easily optimized away. The SharedImageSurface constructor ensures the ImageSurface it receives is unique:

impl SharedImageSurface {
    pub fn new(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        // get_pixel() assumes ARgb32.
        assert_eq!(
            surface.get_format(),
            cairo::Format::ARgb32
        );

        // Ensure the access is unique.
        let reference_count = unsafe {
            cairo_sys::cairo_surface_get_reference_count(
                surface.to_raw_none(),
            )
        };
        assert_eq!(reference_count, 1);

        // Flush any pending drawing operations before
        // accessing the memory directly.
        surface.flush();
        if surface.status() != cairo::Status::Success {
            return Err(surface.status());
        }

        let data_ptr = NonNull::new(unsafe {
            cairo_sys::cairo_image_surface_get_data(
                surface.to_raw_none(),
            )
        }).unwrap();

        let width = surface.get_width();
        let height = surface.get_height();
        let stride = surface.get_stride() as isize;

        Ok(Self {
            surface,
            data_ptr,
            width,
            height,
            stride,
        })
    }
}

And other methods on SharedImageSurface provide access to various surface properties, as well as the pixel data:

impl SharedImageSurface {
    /// Returns the surface width.
    #[inline]
    pub fn width(&self) -> i32 {
        self.width
    }

    /// Returns the surface height.
    #[inline]
    pub fn height(&self) -> i32 {
        self.height
    }

    /// Retrieves the pixel value at the given coordinates.
    #[inline]
    pub fn get_pixel(&self, x: u32, y: u32) -> Pixel {
        assert!(x < self.width as u32);
        assert!(y < self.height as u32);

        let offset =
            y as isize * self.stride + x as isize * 4;
        let ptr = self.data_ptr.as_ptr().offset(offset);

        // According to Cairo documentation, the pixel values
        // for the ARgb32 format should be read using a
        // platform-native u32.
        let value = unsafe { *(ptr as *const u32) };

        Pixel {
            r: ((value >> 16) & 0xFF) as u8,
            g: ((value >> 8) & 0xFF) as u8,
            b: (value & 0xFF) as u8,
            a: ((value >> 24) & 0xFF) as u8,
        }
    }
}

Another nice property of wrapping ImageSurfaces is that it’s possible to provide extra utility methods for working with pixel data. Right now we have methods for extracting the alpha channel, resizing the surface, doing linear sRGB to sRGB and vice versa conversions and performing convolutions useful for primitives like Gaussian blur. Pixel iterators I showcased in the previous post were switched to take SharedImageSurfaces as input. One downside is that any other Cairo APIs that take surfaces as a read-only input need to be wrapped too. Fortunately, so far we needed only the Context::set_source_surface() method which allows using the given surface as the pixel source for drawing operations:

impl SharedImageSurface {
    /// Calls `set_source_surface()` on the given Cairo
    /// context.
    #[inline]
    pub fn set_as_source_surface(
        &self,
        cr: &cairo::Context,
        x: f64,
        y: f64,
    ) {
        cr.set_source_surface(&self.surface, x, y);
    }
}

Converting a SharedImageSurface back into a regular ImageSurface to return it from the filter code does a reference count check too: if it isn’t equal to 1, there are other SharedImageSurfaces pointing at the same ImageSurface, which means we cannot simply return the surface as it can be used for modifying the pixel data thus breaking the SharedImageSurface invariants. In this case a copy of the surface is created and returned:

impl SharedImageSurface {
    /// Converts this `SharedImageSurface` back into a Cairo
    /// image surface.
    #[inline]
    pub fn into_image_surface(
        self,
    ) -> Result<ImageSurface, cairo::Status> {
        let reference_count = unsafe {
            cairo_sys::cairo_surface_get_reference_count(
                self.surface.to_raw_none(),
            )
        };

        if reference_count == 1 {
            Ok(self.surface)
        } else {
            // If there are any other references, copy the
            // underlying surface.
            let bounds = IRect {
                x0: 0,
                y0: 0,
                x1: self.width,
                y1: self.height,
            };

            self.copy_surface(bounds)
        }
    }
}

Alpha-Only Optimizations #

The most heavy filter primitives are the ones doing image convolutions, mainly the Gaussian blur primitive. A convolution involves computing a weighted sum of pixels within a rectangle for every input pixel. The Gaussian blur primitive is frequently used for creating shadows for other elements. In this case it takes the rasterized element’s alpha channel as an input and blurs it. Then the blurred surface is offset a little and drawn below the input surface to create a shadow.

When a convolution is applied to an alpha-only image, there’s no need to compute the weighted sum for the other three color channels. However, going through the whole input image to check if it only contains meaningful data in the alpha channel is rather costly. Thankfully, we can avoid that step altogether by caching this property. I added a field into SharedImageSurface to indicate whether the current surface is alpha-only, along with a special constructor:

struct SharedImageSurface {
    /* ... */

    /// Whether this surface contains meaningful data only
    /// in the alpha channel.
    ///
    /// This is used for optimizations, particularly in
    /// `convolve()` to skip processing other channels.
    alpha_only: bool,
}

impl SharedImageSurface {
    pub fn new(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        /* ... */

        Ok(Self {
            surface,
            data_ptr,
            width,
            height,
            stride,
            // Default to not alpha only.
            alpha_only: false,
        })
    }

    /// Creates a `SharedImageSurface` from a unique
    /// `ImageSurface` with meaningful data only in the alpha
    /// channel.
    #[inline]
    pub fn new_alpha_only(
        surface: ImageSurface,
    ) -> Result<Self, cairo::Status> {
        let mut rv = Self::new(surface)?;
        rv.alpha_only = true;
        Ok(rv)
    }
}

This constructor is used automatically in the extract_alpha() method used to separate the alpha channel of the input surface:

impl SharedImageSurface {
    /// Returns a surface with black background and alpha
    /// channel matching this surface.
    pub fn extract_alpha(
        &self,
        bounds: IRect,
    ) -> Result<SharedImageSurface, cairo::Status> {
        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let output_stride =
            output_surface.get_stride() as usize;
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            for (x, y, Pixel { a, .. }) in
                Pixels::new(self, bounds)
            {
                let output_pixel = Pixel {
                    r: 0,
                    g: 0,
                    b: 0,
                    a,
                };

                output_data.set_pixel(
                    output_stride,
                    output_pixel,
                    x,
                    y,
                );
            }
        }

        // The returned surface is alpha-only!
        SharedImageSurface::new_alpha_only(output_surface)
    }
}

Finally, the pixel processing methods check the alpha-only flag to reduce the number of operations or skip unneeded processing altogether:

impl SharedImageSurface {
    /// Returns a surface with pre-multiplication of color
    /// values undone.
    pub fn unpremultiply(
        &self,
        bounds: IRect,
    ) -> Result<SharedImageSurface, cairo::Status> {
        // Unpremultiplication doesn't affect the alpha
        // channel.
        if self.alpha_only {
            return Ok(self.clone());
        }

        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let stride = output_surface.get_stride() as usize;
        {
            let mut data =
                output_surface.get_data().unwrap();

            for (x, y, pixel) in Pixels::new(self, bounds) {
                data.set_pixel(
                    stride,
                    pixel.unpremultiply(),
                    x,
                    y,
                );
            }
        }

        SharedImageSurface::new(output_surface)
    }

    /// Performs a convolution.
    pub fn convolve(
        &self,
        bounds: IRect,
        target: (i32, i32),
        kernel: &Matrix<f64>,
        edge_mode: EdgeMode,
    ) -> Result<SharedImageSurface, cairo::Status> {
        assert!(kernel.rows() >= 1);
        assert!(kernel.cols() >= 1);

        let mut output_surface = ImageSurface::create(
            cairo::Format::ARgb32,
            self.width,
            self.height,
        )?;

        let output_stride =
            output_surface.get_stride() as usize;
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            if self.alpha_only {
                // Perform a convolution, taking a weighted
                // sum of only the alpha channel.
            } else {
                // Perform a convolution, taking a weighted
                // sum of all four color channels.
            }
        }

        if self.alpha_only {
            SharedImageSurface::new_alpha_only(
                output_surface,
            )
        } else {
            SharedImageSurface::new(output_surface)
        }
    }
}

Validating Performance #

The main motivating example for the optimizations is this old mobile phone SVG file from a 9-year-old issue on performance which takes about 40 seconds to render on my desktop PC:

To measure the performance impact, I built librsvg in release mode with debug info before doing alpha-only optimizations and used the perf tool:

perf record --call-graph=dwarf ./rsvg-convert -o /dev/null mobile_phone_01.svg

I then used the awesome FlameGraph tool to create a nice visualization of the data (click and open in a browser for an interactive SVG):

The large two-part column in the middle happens to be the Gaussian blur, taking up 50.63% of the total rendering time for the mobile phone. Turns out the blur operates on SourceAlpha, which contains just the alpha channel of the rasterized element the filter is applied to. After adding the alpha-only optimizations to convolution and other parts of the code like linear sRGB ⇔ sRGB conversion, the rendering time dropped from 40 seconds to 29 seconds, and the performance graph now looks like this:

The percentage of time taken by Gaussian blur dropped from 50.63% to 36.98%. You can also see a slight drop in the narrow column to the left of the Gaussian blur part: that’s the sRGB linearizations which became no-ops on those input images that were alpha-only.

Conclusion #

The project is coming along very well. I’ve just came back from GUADEC where I really pushed the filter rustification during the hacking days, porting all of the remaining filters from C to Rust. Now that all filters are in Rust and thoroughly tested, I’m looking at improving the performance, starting with the alpha-only optimization described in this post. Also since all SVG nodes are now in Rust (the filter primitives were the last ones), I was able to clean up the remaining C interop code for nodes! It’s great to see the intermediate FFI code gradually disappear as the entire subsystems get completely ported to Rust.

GSoC 2018: Filter Infrastructure

Fri, 08 Jun 2018 07:26:03 +0000

Introduction #

This summer I’m working on librsvg, a GNOME library for rendering SVG files, particularly on porting the SVG filter effects from C to Rust. That involves separating the code for different filters from one huge C file into individual files for each filter, and then porting the filter rendering infrastructure and the individual filters.

Thankfully, in the large C file the code for different filters was divided by comment blocks, so several vim macros later I was done with the not so exciting splitting part.

Representing Filters in Rust #

SVG filter effects are applied to an existing SVG element to produce a modified graphical result. Each filter consists of a number of filter primitives. The primitives take raster images (bitmaps) as an input (this can be, for example, the rasterized element where the filter was applied, the background snapshot of the canvas at the time the filter was invoked, or an output of another filter primitive), do something with it (like move the pixels to a different position, apply Gaussian blur, or blend two input images together) and produce raster images as an output. Each filter primitive has a number of properties. The common properties include the bounds of the region where the filter primitive is doing its processing, the name assigned to the primitive’s result, and the input that the primitive operates on. I collected the common properties into the following types:

struct Primitive {
    x: Cell<Option<RsvgLength>>,
    y: Cell<Option<RsvgLength>>,
    width: Cell<Option<RsvgLength>>,
    height: Cell<Option<RsvgLength>>,
    result: RefCell<Option<String>>,
}

struct PrimitiveWithInput {
    base: Primitive,
    in_: RefCell<Option<Input>>,
}

Each filter primitive struct is meant to contain one of these two common types along with any extra properties as needed. The common types provide functions for parsing their respective properties so that code need not to be duplicated in each filter. Note that these properties are just “descriptions” of the final values to be used during rendering. For example, an RsvgLength can be equal to 2 or 50%, and the actual length in pixels is evaluated during rendering and depends on various rendering state such as the coordinate system in use and the size of the enclosing element. The filter primitive processing behavior is nicely described as a trait:

trait Filter {
    fn render(&self, ctx: &FilterContext)
        -> Result<FilterResult, FilterError>;
}

Here FilterContext contains various filter state such as the rasterized bitmap representation of the SVG element the filter is being applied to and results of previously rendered primitives, and allows retrieving the necessary input bitmaps. Successful rendering results in a FilterResult which has the name assigned to the primitive and the output image, and errors (like non-existent input filter primitive) end up in FilterError. When a filter is invoked, it goes through its child nodes (filter primitives) in order, render()s them and stores the results in the FilterContext.

Pixel Iteration #

Since many filter primitives operate on a per-pixel basis, it’s important to have a convenient way of transforming the pixel values. Librsvg uses image surfaces from Cairo, a 2D graphics library, for storing bitmaps. An image surface stores its pixel values in RGBA format in a large contiguous array row by row with optional strides between the rows. The plain way of accessing the values is image[y * stride + x * 4 + ch] where ch is 0, 1, 2 and 3 for R, G, B and A respectively. However, writing this out is rather tedious and error-prone. As the first step, I added a pixel value struct:

struct Pixel {
    pub r: u8,
    pub g: u8,
    pub b: u8,
    pub a: u8,
}

and extended cairo-rs’s image surface data accessor with the following methods:

fn get_pixel(
    &self,
    stride: usize,
    x: usize,
    y: usize,
) -> Pixel;

fn set_pixel(
    &mut self,
    stride: usize,
    pixel: Pixel,
    x: usize,
    y: usize,
);

using the known trick of declaring a trait containing the new methods and implementing it for the target type. Unfortunately, stride has to be passed through manually because the (foreign) data accessor type doesn’t offer a public way of retrieving it. Adding methods to cairo-rs directly would allow to get rid of this extra argument. Next, since the pattern of iterating over pixels of an image surface within the given bounds comes up rather frequently in filter primitives, I added a Pixels iterator inspired by the image crate. It allows writing code like this:

for (x, y, pixel) in Pixels::new(&image, bounds) {
    /* ... */
}

instead of the repetitive plain version:

for y in bounds.y0..bounds.y1 {
    for x in bounds.x0..bounds.x1 {
        let pixel = image.get_pixel(stride, x, y);
        /* ... */
    }
}

Filters with multiple input images can process pixels simultaneously in the following fashion using the standard Rust iterator combinators:

for (x, y, p, p2) in Pixels::new(&image, bounds)
    .map(|(x, y, p)| {
        (x, y, p, image2.get_pixel(stride, x, y))
    })
{
    let out_pixel = /* ... */;
    out_image.set_pixel(stride, out_pixel, x, y);
}

Benchmarking #

Rust is known for its zero-cost abstractions, however it’s still important to keep track of performance because it’s very well possible to write code in such a way that’s hard to optimize away. Fortunately, a benchmarking facility is provided on nightly Rust out of the box: the test feature with the Bencher type. Benchmark sources are usually placed in the benches/ subdirectory of the crate and look like this:

#![feature(test)]
extern crate rsvg_internals;

#[cfg(test)]
mod tests {
    use super::*;
    use test::Bencher;

    #[bench]
    fn my_benchmark_1(b: &mut Bencher) {
        /* initialization */

        b.iter(|| {
            /* code to be benchmarked */
        });
    }

    #[bench]
    fn my_benchmark_2(b: &mut Bencher) {
        /* ... */
    }

    /* ... */
}

After ensuring the crate’s crate-type includes "lib", you can run benchmarks with cargo +nightly bench. I created three benchmarks, one for the straightforward iteration:

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for y in BOUNDS.y0..BOUNDS.y1 {
        for x in BOUNDS.x0..BOUNDS.x1 {
            let base = y * stride + x * 4;

            r += image[base + 0] as usize;
            g += image[base + 1] as usize;
            b += image[base + 2] as usize;
            a += image[base + 3] as usize;
        }
    }

    (r, g, b, a)
})

One for iteration using get_pixel():

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for y in BOUNDS.y0..BOUNDS.y1 {
        for x in BOUNDS.x0..BOUNDS.x1 {
            let pixel = image.get_pixel(stride, x, y);

            r += pixel.r as usize;
            g += pixel.g as usize;
            b += pixel.b as usize;
            a += pixel.a as usize;
        }
    }

    (r, g, b, a)
})

And one for the Pixels iterator:

b.iter(|| {
    let mut r = 0;
    let mut g = 0;
    let mut b = 0;
    let mut a = 0;

    for (_x, _y, pixel) in Pixels::new(&image, BOUNDS) {
        r += pixel.r as usize;
        g += pixel.g as usize;
        b += pixel.b as usize;
        a += pixel.a as usize;
    }

    (r, g, b, a)
})

Here are the results I’ve got:

test tests::bench_pixels                   ... bench:     991,137 ns/iter (+/- 62,654)
test tests::bench_straightforward          ... bench:     992,124 ns/iter (+/- 7,119)
test tests::bench_straightforward_getpixel ... bench:   1,034,037 ns/iter (+/- 11,121)

Looks like the abstractions didn’t introduce any overhead indeed!

Implementing a Filter Primitive #

Let’s look at how to write a simple filter primitive in Rust. As an example I’ll show the offset filter primitive which moves its input on the canvas by a specified number of pixels. Offset has an input and two additional properties for the offset amounts:

struct Offset {
    base: PrimitiveWithInput,
    dx: Cell<RsvgLength>,
    dy: Cell<RsvgLength>,
}

Since each filter primitive is an SVG node, it needs to implement NodeTrait which contains a function for parsing the node’s properties:

impl NodeTrait for Offset {
    fn set_atts(
        &self,
        node: &RsvgNode,
        handle: *const RsvgHandle,
        pbag: &PropertyBag,
    ) -> NodeResult {
        // Parse the common properties.
        self.base.set_atts(node, handle, pbag)?;

        // Parse offset-specific properties.
        for (_key, attr, value) in pbag.iter() {
            match attr {
                Attribute::Dx => self.dx.set(parse(
                    "dx",
                    value,
                    LengthDir::Horizontal,
                    None,
                )?),
                Attribute::Dy => self.dy.set(parse(
                    "dy",
                    value,
                    LengthDir::Vertical,
                    None,
                )?),
                _ => (),
            }
        }

        Ok(())
    }
}

Finally, we need to implement the Filter trait. Note that render() accepts an additional &RsvgNode argument, which refers to the filter primitive node. It’s different from &self in that it contains various common SVG node state.

impl Filter for Offset {
    fn render(
        &self,
        node: &RsvgNode,
        ctx: &FilterContext,
    ) -> Result<FilterResult, FilterError> {
        // Compute the processing region bounds.
        let bounds = self.base.get_bounds(ctx);

        // Compute the final property values.
        let cascaded = node.get_cascaded_values();
        let values = cascaded.get();

        let dx = self
            .dx
            .get()
            .normalize(&values, ctx.drawing_context());
        let dy = self
            .dy
            .get()
            .normalize(&values, ctx.drawing_context());

        // The final offsets depend on the currently active
        // affine transformation.
        let paffine = ctx.paffine();
        let ox = (paffine.xx * dx + paffine.xy * dy) as i32;
        let oy = (paffine.yx * dx + paffine.yy * dy) as i32;

        // Retrieve the input surface.
        let input_surface =
            get_surface(self.base.get_input(ctx))?;

        // input_bounds contains all pixels within bounds,
        // for which (x + ox) and (y + oy) also lie
        // within bounds.
        let input_bounds = IRect {
            x0: clamp(bounds.x0 - ox, bounds.x0, bounds.x1),
            y0: clamp(bounds.y0 - oy, bounds.y0, bounds.y1),
            x1: clamp(bounds.x1 - ox, bounds.x0, bounds.x1),
            y1: clamp(bounds.y1 - oy, bounds.y0, bounds.y1),
        };

        // Create an output surface.
        let mut output_surface =
            ImageSurface::create(
                cairo::Format::ARgb32,
                input_surface.get_width(),
                input_surface.get_height(),
            ).map_err(FilterError::OutputSurfaceCreation)?;

        let output_stride =
            output_surface.get_stride() as usize;

        // An extra scope is needed because output_data
        // borrows output_surface, but we need to move
        // out of it to return it.
        {
            let mut output_data =
                output_surface.get_data().unwrap();

            for (x, y, pixel) in
                Pixels::new(&input_surface, input_bounds)
            {
                let output_x = (x as i32 + ox) as usize;
                let output_y = (y as i32 + oy) as usize;
                output_data.set_pixel(
                    output_stride,
                    pixel,
                    output_x,
                    output_y,
                );
            }
        }

        // Return the result of the processing.
        Ok(FilterResult {
            name: self.base.result.borrow().clone(),
            output: FilterOutput {
                surface: output_surface,
                bounds,
            },
        })
    }
}

Conclusion #

The project is coming along very nicely with a few simple filters already working in Rust and a couple of filter tests getting output closer to the reference images. I’ll be attending this year’s GUADEC, so I hope to see you there in July!

GSoC 2018: Introduction

Mon, 30 Apr 2018 18:43:40 +0000

Hello! #

I’m Ivan Molodetskikh, a student of the Moscow State University, and I love Rust. I learned about Rust after a couple of years of mainly programming personal projects in C++ (with a little bit of C#, Java and others here and there). Making my way through writing the first Rust project (what’s a better way to get familiar with a language than trying to make something interesting in it?) and trying out various features of the language, I frequently had moments of “This is actually so much better than the C++ alternative!”. Multiple times after inspecting a borrow-related error I realized this has bit me in the past in C++ in a form of a hard to find mistake. Borrowing, lifetimes, iterators, enums, Option and Result in particular are great once you get a feel of how to use them. The Rust community is amazing too with people always being positive and happy to help newcomers. Fast-forward two years, I have a couple of small Rust projects and some contributions and continuing to enjoy the language. So, it should be of no surprise that when I learned about GSoC I started looking for Rust-related projects. I applied to both Xi (a novel text editor with a fully async architecture) and librsvg (a GNOME library for rendering SVG files) and got accepted into librsvg on a project to help with the ongoing effort to port it to Rust, specifically the SVG filter effects.

The project #

Librsvg is a small library for rendering SVG files in the GNOME ecosystem. Its goal is to be a low-footprint library with a minimal API suitable for rendering things like icons and other images that appear on the desktop. Among other things SVG supports the so-called filter effects which generally (but not always) transform existing SVG elements in one way or another. For example, there’s a blur effect, an effect that moves its input by a fixed position, and an effect that loads an external raster image. Currently all filter effects in librsvg are implemented entirely in C. Additionally, most of them aren’t covered by specification conformance tests. For this project I’m going to be porting the filter infrastructure to Rust, adding all missing tests and making sure everything works correctly. There’s a number of things to get done as part of the project. I’ll start by separating the existing C filter code from one huge file into multiple small ones, one for each filter. This will make it easier to port filters to Rust one by one later. Next comes the most interesting part, experimenting with Rust abstractions over common filter actions, such as iterating over pixels in various ways, like one by one or using a square window. This has to be fast and ergonomic and support the different filter use cases. Finally, I’ll be porting all filter code from C to Rust and adding the missing filter tests along the way to make sure everything works right.

Conclusion #

I’ll be posting about my progress on this blog. If you want to contact me, I’m always in #rust on the GNOME IRC. And if you’re into video game speedruns, this summer I’ll be at the ESA hanging out and helping to commentate the Half-Life run. 🙂