Skip to content

Add extern "custom" #3980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
folkertdev wants to merge 6 commits into
base: master
Choose a base branch
from
+168 −0
Open

Add extern "custom" #3980

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bc0c28d
add `extern "custom"` RFC
folkertdev Jul 1, 2026
b38e056
changes after review
folkertdev Jul 1, 2026
231b22f
Apply suggestions from code review
folkertdev Jul 2, 2026
eba79be
Fix abi_custom header levels
ehuss Jul 1, 2026
5573cea
Rename 3980 extern-custom
ehuss Jul 1, 2026
206ea92
Set 3980 RFC PR link
ehuss Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions text/3980-extern-custom.md
View file
Open in desktop

@Jules-Bertholet Jules-Bertholet Jul 7, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are extern "custom" function pointers allowed? The current implementation on nightly considers unsafe extern "custom" fn(), extern "custom" fn(), and extern "custom" fn(i32) -> u64 to all be valid types.

One option would be to have a single bare extern "custom" function pointer type, to represent a function with known address but unknown calling convention. All other function pointers would be able to coerce to it.

View changes since the review

@steffahn steffahn Jul 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! IMHO, it would make sense if you can only define function items as extern "custom" when they're unsafe, argument-less and returning () (or !), to apply the same kind of restrictions on what fn pointer types are usable.

@steffahn steffahn Jul 8, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m looking at RFC 2972 constrained_naked - specifically the future possibilities section - and I think that it’s food for thought on this design. Here’s a quote of that section:

Future possibilities

It would be possible to define new calling conventions that can be used with naked functions.

A previous version of this document defined an extern "custom" calling convention. It was observed in conversation that calling conventions are really a type and that it could be useful to have calling conventions as part of the type system. In the interest of moving forward with constrained naked functions, it is best to limit the scope of this RFC and defer this (very good) conversation to a future RFC. As a simple workaround, naked functions which do not conform to their specified calling convention should be marked as unsafe and the caller requirements should be documented in the safety section of the documentation per standard convention.

It may also be possible to loosen the definition of a naked function in a future RFC. For example, it might be possible to allow the use of some additional, possibly new, operands to the asm!() block.

In particular the middle part

It was observed in conversation that calling conventions are really a type and that it could be useful to have calling conventions as part of the type system. In the interest of moving forward with constrained naked functions, it is best to limit the scope of this RFC and defer this (very good) conversation to a future RFC. As a simple workaround, naked functions which do not conform to their specified calling convention should be marked as unsafe and the caller requirements should be documented in the safety section of the documentation per standard convention.

seems very relevant IMHO to the discussion of allowing arguments, or allowing non-unsafe signatures to the function. I get the argument that there’s a distinction between cases where it’s just the ABI that’s “special”, but other than that the function is safe to call – from the cases where the function has additional safety predicates.

But I also don’t think this is really how Rust works. If there are still manually-upheld preconditions (not expressed in the type signature) then it simply must be unsafe; no compromises on that: The ABI being unspecified still means that the whole thing is inherently unsafe to use.

I believe that to really allow a "custom" ABI function (and consequently also function pointer type) to call itself truly safe (i.e. non-unsafe), it would need to indicate the ABI at the type level somehow.

I also believe that allowing function arguments would - practically - already allow a way to do this. People could establish a convention where the first argument to a extern "custom" fn(…) is actually a marker type that identifies the actual ABI. If you define your own ABI (by “define”, here I just mean documenting it, in the docs for the MyAbi marker type itself), then function pointers like

let f: extern "custom" fn(MyAbi, x: u32, y: u32) -> u32;

could be sensibly be considered fully safe. They could be safe in the sense that you can then provide a sound way to call them from safe code, e.g. as a callback:

fn call_my_abi(callback: extern "custom" fn(MyAbi, u32, u32) -> u32, x: u32, y: u32) -> u32 { … }

fn main() {
    let f: extern "custom" fn(MyAbi, x: u32, y: u32) -> u32 = todo!();
    let n: u32 = call_my_abi(f, 42, 123); // yay, no unsafe!
}

I also think that establishing such a convention is somewhat questionable for downstream users to do (without any official “blessing” of the practice), since different conventions could be incompatible.1 Furthermore, if we ever wanted to have a different, more “dedicated” mechanism for specifying custom ABIs on a type level in the future2, then it’d be incompatible with this kind of approach. I personally wouldn’t dislike the idea of just blessing this kind of approach though, and to leave any sort of better support for it up to future possibilities.

Speaking of future possibility: I could even imagine a trait like trait CustomAbi<Args, Output> { … }3 that would allow you to somehow define how to use an function of that ABI (on the argument and/or return types that the ABI supports) from Rust. So you can just call foo(x, y) on an extern "custom" fn foo(MyAbi, u32, u32) -> u32 function (or on a function pointer), provided there exists some implementation impl CustomAbi<(u32, u32), u32> for MyAbi.

Alternatively, if we don’t want to discuss any of this at this point, it would in my opinion make a lot of sense to uphold the restriction of only unsafe function for now - in order not to close this kind of future possibility.

View changes since the review

Footnotes

  1. Imagine one convention is to put a marker type as first argument, another is to put them as last argument, and someone defines some ABIs where basically any Rust type could be handled by passing a pointer - then I can imagine overall-unsound APIs emerging once you create some extern "custom" fn(AbiConvention1, OtherArg, AbiConvention2) that fits both conventions. ↩

  2. Something like extern "custom(MyAbi)" fn(u32, u32) -> u32 perhaps? Or extern "custom"(MyAbi) fn(u32, u32) -> u32, i.e. with somewhat more dedicated syntax to keep the type outside of the string literal? ↩

  3. The question of what exactly goes inside of this trait is slightly tricky since there’s currently no way to write something like “extern "custom" fn(Self, Args…) -> Output” as a type, that “unpacks” a tuple Args into function arguments. ↩

@programmerjake programmerjake Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question of what exactly goes inside of this trait is slightly tricky since there’s currently no way to write something like “extern "custom" fn(Self, Args…) -> Output” as a type, that “unpacks” a tuple Args into function arguments.

There's currently an experimental implementation of fn f(..., #[splat] args: ArgsTuple), tracking issue: rust-lang/rust#153629

Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
- Feature Name: `abi_custom`
- Start Date: 2026-07-01)
- RFC PR: [rust-lang/rfcs#3980](https://github.com/rust-lang/rfcs/pull/3980)
- Rust Issue: [rust-lang/rust#140829)](https://github.com/rust-lang/rust/issues/140829)

## Summary
[summary]: #summary

An `extern "custom" fn` is a function with a custom ABI that is unknown to rust. Often these are low-level functions that pass arguments in different registers than any standard calling convention, so using this helps rustc block you from using this in any place where rustc would need to understand that calling convention.


```rust
/// # SAFETY
///
/// - Expects the dividend and the divisor in r0 and r1.
/// - Returns the quotient in r0 and the remainder in r1.
#[unsafe(naked)]
pub unsafe extern "custom" fn __aeabi_uidivmod() {
Comment thread
folkertdev marked this conversation as resolved.

@Jules-Bertholet Jules-Bertholet Jul 5, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub unsafe extern "custom" fn __aeabi_uidivmod() {
pub unsafe extern "custom" fn __aeabi_uidivmod {

Since there's no parameter list, and you can't use a call expression with an extern "custom" function, arguably there should be no parentheses in the declaration. Not sure it's worth changing the language grammar for this, though.

Another option is to use the variadic dots syntax:

Suggested change
pub unsafe extern "custom" fn __aeabi_uidivmod() {
pub unsafe extern "custom" fn __aeabi_uidivmod(...) {

View changes since the review

@kennytm kennytm Jul 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing the () breaks the Function grammar, absolutely not worth it for a pretty niche feature.

making it (...) could work, though i like #3980 (comment) more.

@zachs18 zachs18 Jul 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 for removing the parens entirely, seems not worth changing the grammar for.

+0.5 for ..., but I'd prefer just allowing arguments and return types for documentation reasons.

@bjorn3 bjorn3 Jul 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variadic arguments are passed differently from regular arguments on some calling conventions though.

@folkertdev folkertdev Jul 5, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is a niche feature, that almost nobody will see much less use, I see no reason to spend any syntax budget here.

it's funny/frustrating how so many RFC threads devolve into discussions about syntax: everyone can have an opinion about syntax. The bar for adding syntax is, very deliberately, extremely high.

@steffahn steffahn Jul 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@folkertdev it seems to be human nature or sth. Wadler's law in full effect 😄

core::arch::naked_asm!(
"push {{lr}}",
"sub sp, sp, #4",
"mov r2, sp",
"bl {trampoline}",
"ldr r1, [sp]",
"add sp, sp, #4",
"pop {{pc}}",
trampoline = sym crate::arm::__udivmodsi4
);

@Jules-Bertholet Jules-Bertholet Jul 7, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth it to allow the naked_asm! blocks in extern "custom" functions to specify clobbered registers? This could be useful for documentation.

If we allow an explicit parameter list as suggested in #3980, perhaps we could even allow specifying inputs and outputs that reference those parameters (and also the return place?)

View changes since the review

}

unsafe extern "custom" {
fn __fentry__();
}
```

### History

* Proposal: https://github.com/rust-lang/rust/issues/140566
* Tracking issue: https://github.com/rust-lang/rust/issues/140829
* Implementation: https://github.com/rust-lang/rust/pull/140770
* Stabilization PR: https://github.com/rust-lang/rust/pull/158504

## Motivation
[motivation]: #motivation

In some low-level scenarios we must define naked functions with a custom ABI. This comes up in `rust-lang/compiler-builtins` (e.g. for `__rust_probestack` and `__aeabi_uidivmod`), and also in systems programming (e.g. when defining `__fentry__`, a symbol used the mcount mechanism).

The current solution is often to use `extern "C"`, but that is misleading: the function does not use the C calling convention, and calling it as if it were is almost certainly causing UB.

## Guide-level explanation
[guide-level-explanation]: #guide-level-explanation

Because rust doesn't know what calling convention to use, an `extern "custom"` function can only be called via inline assembly or FFI.

```
error: functions with the "custom" ABI cannot be called
--> <source>:5:5
|
5 | bar();
| ^^^^^
|
note: an `extern "custom"` function can only be called using inline assembly
```

An `extern "custom"` function definition must be a naked function:

```
error: items with the "custom" ABI can only be declared externally or defined via naked functions
--> <source>:10:1
|
10 | unsafe extern "custom" fn bar() {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
help: convert this to an `#[unsafe(naked)]` function
|
10 + #[unsafe(naked)]
11 | unsafe extern "custom" fn bar() {
|
```

An `extern "custom"` function definition must be unsafe. The intent here is that a safety comment is written on how this function may be called.

```
error: functions with the "custom" ABI must be unsafe
--> <source>:10:1
|
10 | extern "custom" fn bar() {
| ^^^^^^^^^^^^^^^^^^^^^^^^
|
help: add the `unsafe` keyword to this definition
|
10 | unsafe extern "custom" fn bar() {
| ++++++
```
Comment on lines +81 to +94

@Darksonn Darksonn Jul 2, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since all ways of actually calling an extern "custom" function are unsafe, it doesn't seem required that we also make the functions themselves unsafe. Yes, you must call it with the right ABI for the call to be safe, but that's true for all functions. Defining such a function just seems analogous to creating a raw pointer to me.

View changes since the review

@clarfonthey clarfonthey Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The justification to me seems to be to encourage people to write safety comments describing the actual ABI, but I actually kind of agree and think we could have a separate clippy lint for documenting calling conventions for extern "custom" methods.

Also from the perspective of, should these function definitions be allowed with #[deny(unsafe_code)], I think the answer is yes, since only calling them is unsafe.

@kennytm kennytm Jul 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't seem required that we also make the functions themselves unsafe.

If an extern "custom" fn can be defined as a safe function, it follows that they should also be able to be declared as safe, invalidating lines 96-109 (In an extern "custom" block, functions cannot be marked as safe).

The justification to me seems to be to encourage people to write safety comments describing the actual ABI

Maybe that mandatory #[unsafe(naked)] is enough for encouragement

should these function definitions be allowed with #[deny(unsafe_code)], I think the answer is yes, since only calling them is unsafe.

But this argument applies to defining ordinary unsafe functions too, yet this will fire the lint

#[expect(unsafe_code)]
unsafe fn just_defining_a_function_not_calling_it() {}

So unless we further refine unsafe_code into unsafe_use & unsafe_define, this should continue to be denied.

@Lokathor Lokathor Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we change the unsafe requirements, lines 96-109 would also be changed. That's not a strong justification either way.

@folkertdev folkertdev Jul 2, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe that mandatory #[unsafe(naked)] is enough for encouragement

The unsafe on naked is there to declare that the body faithfully implements the signature. Hence standard naked functions can in fact be safe to call, despite their implementation being inherently unsafe due to the use of inline assembly.

@Darksonn Darksonn Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I guess for extern "custom" functions, whether or not it's an unsafe fn is just as meaningless as which arguments appear in its arguments list. The only real difference is how people will interpret the function when they read it.

For instance, using __aeabi_uidivmod from the original motivation:

/// Multiplies two 64-bit integers.
///
/// # ABI
///
/// The first argument is passed across registers r0 (lower 32 bits) and r1
/// (upper 32 bits). The second argument spans registers r2 and r3. It returns
/// the final 64-bit result across r0 and r1.
#[naked]
pub extern "custom" fn __aeabi_lmul(multiplier: u64, multiplicand: u64) -> u64 {
    ...
}

/// Divides two 32-bit integers, computing both the quotient and remainder.
///
/// # ABI
///
/// The numerator (top number) and denominator (bottom number) are passed into
/// CPU registers r0 and r1. It returns the quotient in register r0 and the
/// remainder in register r1.
///
/// # Safety
///
/// The divisor must be non-zero.
#[naked]
pub unsafe extern "custom" fn __aeabi_uidivmod(numerator: u32, denominator: u32) -> (u32, u32) {
    ...
}

Here, we make __aeabi_lmul safe because it's safe for all inputs, and we make __aeabi_uidivmod unsafe because it's not safe for all inputs. In either case, we also document its ABI, and to actually call it with inline asm you must of course follow that ABI, but that's a separate matter from the function's own safety annotation.

Of course, neither the argument lists or whether it's marked unsafe has any real effect here except insofar it helps the reader understand the function.

@poseidon-rises poseidon-rises Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3980 (comment)

I'll remake the argument above here, these annotations make the functions easier for a reader who just wants to know what some code it doing, and doesn't care about the specifics of the ABI. I think, since it has no effect on the usage of the function, I see no reason to disallow it or add special handling just to deny safe custom ABIs.


In an `extern "custom"` block, functions cannot be marked as `safe`:

```
error: foreign functions with the "custom" ABI cannot be safe
--> <source>:16:5
|
16 | safe fn foobar();
| ^^^^^^^^^^^^^^^^^
|
help: remove the `safe` keyword from this definition
|
16 - safe fn foobar();
16 + fn foobar();
```

An `extern "custom"` function cannot have any arguments or a return type:

```
error: invalid signature for `extern "custom"` function
--> <source>:6:31
|
6 | unsafe extern "custom" fn foo(a: i32) -> i32 {
| ^^^^^^ ^^^
Comment on lines +111 to +118

@kennytm kennytm Jul 2, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the current implementation in 1.98.0-nightly (2026-07-01 4c9d2bfe4ad7a6566909) returning ! seems to be accepted (but returning an actual never_type i.e. (!) or std::convert::Infallible is rejected). please clarify

#![feature(abi_custom, never_type)]

use std::arch::naked_asm;

#[unsafe(naked)]
unsafe extern "custom" fn foo() -> ! {   // no error.
    naked_asm!("ud2")
}

#[unsafe(naked)]
unsafe extern "custom" fn bar() -> (!) { // error: invalid signature for `extern "custom"` function
    naked_asm!("ud2")
}

View changes since the review

@clarfonthey clarfonthey Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just feels like a bug IMHO, since no ABI means no way to tell if the function never returns. That said, technically, since crates using this method would benefit from knowing this, perhaps it is a reason to allow it, perhaps via some unsafe(no_return) attribute or something similar.

@scottmcm scottmcm Jul 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, is there a reason it needs to care that you wrote arguments? Since you can't call it anyway, would it make more sense to just say "sure, write whatever arguments is helpful" and rust will just not use them for anything (other than rustdoc)?

@poseidon-rises poseidon-rises Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @scottmcm here, since we can't make a direct call, why should we limit the signature? I don't see any disadvantages to adding arbitrary parameters and return types, but personally, I find it easier to parse __aeabi_uidivmod(numerator: u32, denominator: u32) -> (u32, u32) at a glance over __aeabi__uidiv_mod(). Specifically, I don't need to read the exact ABI to get a general gist of the purpose of this function.

@Lokathor Lokathor Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, the function still has semantic inputs and outputs, even if the compiler doesn't know the ABI for them, so we should prefer that the function show those inputs and outputs even if only as documentation.

|
= note: functions with the "custom" ABI cannot have any parameters or return type
help: remove the parameters and return type
|
6 - unsafe extern "custom" fn foo(a: i32) -> i32 {
6 + unsafe extern "custom" fn foo() {
|
```

## Reference-level explanation
[reference-level-explanation]: #reference-level-explanation

See https://github.com/rust-lang/reference/pull/2300.

## Drawbacks
[drawbacks]: #drawbacks

No specific drawback.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives

https://github.com/rust-lang/rust/issues/140566

The name has already been debated. The name `unknown` has been mentioned, but to write the implementation you really do need to know the ABI. It is custom in the sense that rustc does not know about it, but the author definitely does.

Comment thread
folkertdev marked this conversation as resolved.
Arguments and return types are forbidden because they are not consumed in any way. The compiler has no way of validating them against the function body, similar to other naked functions, but the functions can never be invoked directly so they serve no purpose when calling. The intent is that correct parameter passing and returning will be covered in documentation.

## Prior art
=======
[prior-art]: #prior-art

https://github.com/rust-lang/rust/issues/140566#issuecomment-2846205457

We do have some other calling conventions that cannot be called using rust's function calling syntax:

- extern "*-interrupt`
- extern "gpu-kernel"

Those however do have a known ABI, it's just that semantically it does not make sense to call them from a rust program.

## Unresolved questions
[unresolved-questions]: #unresolved-questions

None currently.

## Future possibilities
[future-possibilities]: #future-possibilities

None currently.