Cancellation Safety in different Rust AsyncRuntimes

2026-04-01T00:00:00+00:00

To get more context I recommend my dear friends to read or watch a great talk</a> made by Rain, where they/she described her attitude to “cancellation safety” in async Rust.</p>

TL;DR for those who prefer to hack and not to read:</p>

git clone https://gitlab.com/blogging1/cancellation-safety</code>;</li>

cargo test --no-fail-fast --workspace</code> (or your runtime of choice by -p cancel_%RUNTIME%</code> intsead of --workspace</code>);</li>

Read errors and tests;</li>

Read lib.rs</code> for runtimes you are interested in;</li>

Play with the code to fix all the tests;</li>
</ul>

</i></a>
The Problem</h1>
“Cancellation Safety” usually means absence of unexpected “effects” when sequence of asynchronous operations has been stopped
before it reaches the final state.
I can think of a panic!</code> in the middle of a synchronous sequence of operations as an analogy.</p>
The fundamental principles of it tied to the Future</code> trait and to the implementation details of the Asynchrnous Runtime.
Here, I will like to experiment in an attempt to find common patterns and differences in how Tokio, smol and glommio handle cancellations.</p>

Future::poll</code> is a key aspect here, as at compile-time every Future</code> implementation is a state-machine, that represents a sequence of operations,
“cancellation” means that Runtime</strong> (Reactor, Event Loop or Scheduler) will stop polling this state-machine and (hopefully) destroy it’s state;</li>
Pin</code> and Unpin</code> have their own places, as !Unpin</code> types have to keep additional guarantees to be considered “safe” (mitigate “effects”);</li>
Waker</code> implementation may play a big role in handling “cancellation” requests;</li>
</ul>
</i></a>
The Example</h1>
In our adventure we will go join our strong and proud friends who lives in Moria:</p>
pub</span> async</span> fn</span> mine_with_tool</span><</span>F</span>,</span> FT</span>></span>(</span>dwarf</span>:</span> &</span>mut</span> Dwarf</span>,</span> mut</span> pickaxe</span>:</span> F</span>)</span> ...</span> {</span></span>
    let</span> mut</span> bag</span>:</span> Bag</span> =</span> Vec</span>::</span>new</span>(</span>)</span>;</span></span>
    for</span> i</span> in</span> 0</span>..</span>MAX_ALLOWED_SHIFTS</span> {</span></span>
        pickaxe</span>(</span>)</span>.</span>await</span>;</span></span>
        ...</span></span>
        println!</span>(</span>"</span>Here at the Gates the king AWAITS</span>"</span>)</span>;</span></span>
        ...</span></span>
        bag</span>.</span>push</span>(</span>Ferrum</span>::</span>Dirty</span>)</span>;</span></span>
    }</span></span>
    dwarf</span>.</span>bag </span>=</span> Some</span>(</span>bag</span>)</span>;</span></span>
}</span></span></code></pre>
so our asynchronous process will imitate dwarf working in mines. The mine_with_tool</code> itself is a composite trait Future</code> implementation,
that internally do MAX_ALLOWED_SHIFTS</code> steps to reach it’s final (completed) state.
In simple terms, line 4: pickaxe().await;</code> will be a point when state-machine polled and make progress. It means that this exact spot
may be used to “cancel” the execution.</p>
How it loolks at high-level intermediate representation (code cleaned to make it less verbose, please run cargo +nightly rustc -- -Z unpretty=hir</code>
inside moria</code> folder to see the full output):</p>
async</span> fn</span> mine_with_tool</span><</span>F</span>,</span> FT</span>></span>(</span>dwarf</span>:</span> &</span>'</span>_</span> mut</span> Dwarf</span>,</span> pickaxe</span>:</span> F</span>)</span> -></span> /*</span>impl Trait</span>*/</span></span>
    where</span> F</span>:</span> FnMut</span>(</span>)</span> -></span> FT</span>,</span> FT</span>:</span> std</span>::</span>future</span>::</span>Future</span> |</span>mut</span> _task_context</span>:</span> ResumeTy</span>|</span></span>
{</span></span>
    ...</span></span>
    let</span> mut</span> bag</span>:</span> Bag</span> =</span> Vec</span>::</span>new</span>(</span>)</span>;</span></span>
    {</span></span>
        ...</span></span>
        loop</span> {</span></span>
            match</span> next</span>(</span>&</span>mut</span> iter</span>)</span> {</span></span>
                None</span> {</span>}</span> =></span> break</span>,</span></span>
                Some</span> {</span>  0</span>:</span> i</span> }</span> =></span> {</span></span>
                    match</span> into_future</span>(</span>pickaxe</span>(</span>)</span>)</span> {</span></span>
                        mut</span> __awaitee</span> =></span></span>
                            loop</span> {</span></span>
                                match</span> unsafe</span> {</span></span>
                                        poll</span>(</span>new_unchecked</span>(</span>&</span>mut</span> __awaitee</span>)</span>,</span></span>
                                            get_context</span>(</span>_task_context</span>)</span>)</span></span>
                                    }</span> {</span></span>
                                    Ready</span> {</span>  0</span>:</span> result</span> }</span> =></span> break</span> result</span>,</span></span>
                                    Pending</span> {</span>}</span> =></span> {</span> }</span></span>
                                }</span></span>
                                _task_context</span> =</span> (</span>yield</span> (</span>)</span>)</span>;</span></span>
                            }</span>,</span></span>
                    }</span>;</span></span>
                    ...</span></span>
                    bag</span>.</span>push</span>(</span>Ferrum</span>::</span>Dirty</span>)</span>;</span></span>
                }</span></span>
            }</span></span>
        }</span>,</span></span>
        ...</span></span>
    dwarf</span>.</span>bag </span>=</span> Some</span>(</span>bag</span>)</span>;</span></span>
    ...</span></span>
}</span></span></code></pre>
alomost no “async” magic anymore, we have a loop</code>, an unsafe</code> and a yield ()</code>.</p>
Let’s jump to a “safety” aspect. In my humble opinion, if there is no unsafe</code> code and Rust compiler is pleased, the operation is safe.
Yet, it doesn’t mean that the program behaves the way programmers may expect it to behave.</p>
...</span> fn</span> mine_with_tool</span><</span>...</span>></span>(</span>dwarf</span>:</span> &</span>mut</span> Dwarf</span> ...</span> {</span></span>
    let</span> mut</span> bag</span>:</span> Bag</span> =</span> Vec</span>::</span>new</span>(</span>)</span>;</span></span>
        ...</span></span>
        bag</span>.</span>push</span>(</span>Ferrum</span>::</span>Dirty</span>)</span>;</span></span>
        ...</span></span>
    dwarf</span>.</span>bag </span>=</span> Some</span>(</span>bag</span>)</span>;</span></span>
}</span></span></code></pre>
try to spot yourself what makes this trait Future</code> implementation be named “cancellation not-safe” (I prefer not-safe here, as unsafe</strong> is important
but unrelated Rust term).</p>
The answer lies in how this asynchronous operation stores it’s state and tracks the own progress. Both things happen internally, while holding a mutable
reference to the state given by the caller side dwarf: &mut Dwarf</code>. Let’s see two examples of this feature usage inside smol</code> runtime to see why
such implementation can give a surprises to caller if cancelled.</p>
pub</span> async</span> fn</span> work</span>(</span>dwarf</span>:</span> &</span>mut</span> Dwarf</span>)</span> {</span></span>
    super</span>::</span>mine</span>(</span>dwarf</span>)</span></span>
        .</span>or</span>(</span>async</span> {</span></span>
            Timer</span>::</span>after</span>(</span>HALF_SHIFT</span>)</span>.</span>await</span>;</span></span>
        }</span>)</span></span>
        .</span>await</span>;</span></span>
}</span></span></code></pre>
Dwarf works here half of the shift time and made some progress.
But when we go to see run the test:</p>
let</span> mut</span> dwallin</span> =</span> Dwarf</span>::</span>new</span>(</span>Name</span>::</span>Dwalin</span>)</span>;</span></span>
timeout</span>::</span>work</span>(</span>&</span>mut</span> dwallin</span>)</span>.</span>await</span>;</span></span>
//</span> Bag is empty, but I heard the song!</span></span>
assert!</span>(</span>dwallin</span>.</span>bag</span>.</span>is_some</span>(</span>)</span>)</span>;</span></span></code></pre>
we see it’s assertation that some work has been done fails. The reason is that the timeout happened before Dwallin was able to finish
his work and all what he put into the “bag” in internal state of the state-machine has been dropped once we cancel the operation at the
first poll after timeout. (Try to change Timer::after(HALF_SHIFT)</code> to use FULL_SHIFT</code> and see if it will help 😉).
So this is what we may say a cancellation not-safe (or incorrect as Rain proposed) implementation of asynchronous operation that leads to
a behaviour that we would not expect.</p>
More interesting (and closer to real-world) can happen if we make our Future</code> implementation more complex and “dirty”:</p>
pub</span> async</span> fn</span> work</span>(</span>dwarf</span>:</span> &</span>Mutex</span><</span>Dwarf</span>></span>)</span> {</span></span>
    let</span> mut</span> dwarf_guard</span> =</span> dwarf</span>.</span>lock</span>(</span>)</span>.</span>await</span>;</span></span>
    let</span> mut</span> old_bag</span> =</span> dwarf_guard</span>.</span>bag</span>.</span>take</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
    timeout</span>(</span>HALF_SHIFT</span>,</span> super</span>::</span>mine</span>(</span>&</span>mut</span> dwarf_guard</span>)</span>)</span></span>
        .</span>await</span></span>
        .</span>err</span>(</span>)</span></span>
        .</span>unwrap</span>(</span>)</span>;</span></span>
    if</span> let</span> Some</span>(</span>new_bag</span>)</span> =</span> dwarf_guard</span>.</span>bag</span>.</span>as_mut</span>(</span>)</span> {</span></span>
        new_bag</span>.</span>append</span>(</span>&</span>mut</span> old_bag</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
I know, Dwarfs are not good at asynchronous programming, but it illustrates the problem really good.
Our operation takes state out of input, holds it internally, than accumulates with own computation results and give it back.</p>
┌────────────────────────────────────────────────────────────────────────────────────────┐</span></span>
│  ┌────────────┐    ┌──────────────┐    ┌───────┐    ┌─────────────┐    ┌──────────────┐│</span></span>
│  │ Lock Mutex │ -> │ Take Old Bag │ -> │ Mine  │ -> │Take New Bag │ -> │  Merge Bags  ││</span></span>
│  └────────────┘    └──────────────┘    └───────┘    └─────────────┘    └──────────────┘│</span></span>
│  ┌────────────┐    ┌──────────────┐                                                    │</span></span>
│->│Release Lock│ -> │    End       │                                                    │</span></span>
│  └────────────┘    └──────────────┘                                                    │</span></span>
└────────────────────────────────────────────────────────────────────────────────────────┘</span></span></code></pre>
The issue is that operation is “safe” from the mutex point of view, as we know noone else will change the state of the dwarf</code>.
Yet it is not “correct” if we cancel super::mine</code> before it fully completes. old_bag</code> will be dropped as new_bag</code> will not
be a thing (dwarf_guard.bag.as_mut()</code> is None</code>). As work</code> future implementation is a composition of itself with
super::mine</code> future implementation and timeout</code> future implementation, our logic becomes broken, because both futures
have incorrect behavior from the system perspective (while it is totally expected, valid and safe</strong> Rust code imho).</p>
</i></a>
The Comparison</h1>
</i></a>
Runtime Architecture Overview</h2>
</i></a>
Tokio: The Industry Standard</h3>
Tokio represents the most widely used async runtime in the Rust ecosystem. It’s architecture focuses on:</p>

Multi-threaded thread pool executor</strong> for CPU-bound work;</li>
Completer-based I/O model</strong> for non-blocking operations;</li>
Robust task cancellation</strong> with explicit abort capabilities;</li>
Comprehensive ecosystem</strong> of supporting libraries;</li>
</ul>
Key characteristics:</p>

Built-in cancellation tokens via tokio_util::sync::CancellationToken</code>;</li>
tokio::spawn()</code> for creating tasks;</li>
tokio::select!</code> for concurrent operations;</li>
Built-in timeout utilities;</li>
</ul>
</i></a>
smol: The Minimal Approach</h3>
smol takes a radically different approach with:</p>

Single-threaded executor</strong> by default;</li>
Simplified API</strong> focusing on essential async operations;</li>
No own runtime</strong> - it brings to you existing executors;</li>
Lightweight dependencies</strong> and minimal overhead;</li>
</ul>
Key characteristics:</p>

smol::spawn()</code> for task creation;</li>
smol::Timer</code> for async delays;</li>
smol::channel</code> for message passing;</li>
No explicit cancellation tokens in core API;</li>
</ul>
</i></a>
Glommio: I/O Performance Focus</h3>
Glommio represents a specialized runtime designed for high-performance I/O workloads:</p>

Local executor model</strong> with share-nothing-first approach;</li>
I/O-optimized</strong> with dedicated thread-per-core architecture;</li>
No shared memory</strong> between threads by default;</li>
Local-only futures</strong> for better cache locality;</li>
</ul>
Key characteristics:</p>

glommio::LocalExecutor</code> for single-threaded execution;</li>
glommio::spawn_local()</code> for tasks;</li>
glommio::timer</code> for delays;</li>
glommio::channels::local_channel</code> for local-only communication;</li>
</ul>
</i></a>
Cancellation Scenarios Analysis</h2>
Most of the scenarios are work similar across runtimes:</p>

Simple Drop Cancellation: Use explicit drop mechanisms for futures and all runtimes show partial work loss when futures are dropped unexpectedly;</li>
Timeout Cancellation: All provide explicit timeout handling and tests show that timeout doesn’t preserve partial results;</li>
Mutex-Protected Operations: tokio::sync::Mutex</code>, smol::lock::Mutex</code>, glommio::sync::RwLock</code> all share similar semantics,
while tests demonstrate lock holding patterns and cleanup works the way they should, while we still loss our progress;</li>
Channel-Based Communication: very similar, only with locality nuances;</li>
</ul>
But some of them have nuances:</p>

Macroses like select!</code> and Concurrent Operations with Cancellation Tokens are specific to Tokio</code> runtime ecosystem, which seems to be not a good
or bad things. As Rain described in her talk and what I heard from other developers, such macroses sometimes are totally banned in projects due to
their non-explicit nature (and for example replaced with futures_concurrency</a>);</li>
“Explicit” Cancel:

in Tokio</code> is done via JoinHandle::abort</code> and dropping handle will not cause a cancellation, nahdle.await</code> will return an error JoinError</code>
if future implementation was cancelled (or normal result if it finished), also worth noting that spawn_blocking</code> tasks are not “cancellable” because they are not asynchronous (but it will prevent the task from starting if it wasn’t yet!);</li>
in smol</code> you can call Task::cancel</code> and wait for the cancellation (which may return Some</code> if it finished), it is similar to dropping the future implementation;</li>
in glommio</code> approach is also identical to smol</code> with Option</code> being returned on awaiting of the cancellateion;</li>
</ul>
</li>
</ul>
From the documentation side this topic is only covered in Tokio</code> API docs, smol</code> has one small (ha-ha) note on this matter:Note that canceling a task actually wakes it and reschedules one last time. Then, the executor can destroy the task by simply dropping its Runnable or by invoking run().</strong>, while in glommio</code> I was unable to find mentioning cancellation safety or correctness. I think there are two factors why it is, what it is:</p>

Tokio has much bigger popularity and usage, though have more resources to add documentation;</li>
Tokio has much “dangerous” API and internal executor model that makes it easier to run into cancellation safety issues using it;</li>
</ol>
I hope you found something new in this blog post, write your thoughts in the comments and check additional resources if you want to.</p>
</i></a>
Additional Resources and Sources of inspiration</h1>

Great Rain’s talk</a>;</li>
Current comment on state of the book section</a> about cancellation safety;</li>
Tokio Docs on that matter</a></li>
Sructured Concurrency lib</a></li>
</ul>

Bishonen's Blog - Rust

Cancellation Safety in different Rust AsyncRuntimes

Rust Types - Taxonomy

Okxlivro - Post Mortem

Rust Microservices - Project Structure

Bishonen's Blog - Rust

Cancellation Safety in different Rust AsyncRuntimes

</i></a> smol: The Minimal Approach</h3> smol takes a radically different approach with:</p> Single-threaded executor</strong> by default;</li> Simplified API</strong> focusing on essential async operations;</li> No own runtime</strong> - it brings to you existing executors;</li>

Rust Types - Taxonomy

Okxlivro - Post Mortem

Rust Microservices - Project Structure