Selectively Enabling Execution Platforms

[katre]

Execution platforms are currently either registered or not registered, as opposed to toolchains, which can be selectively enabled via the target_settings attribute.

This proposal is to add a new attribute on platform, called enabled_settings, to perform the same purpose.

See the full proposal here: https://github.com/bazelbuild/proposals/blob/main/designs/2024-04-26-platform-exec-settings.md

This is based on my previous question about executions platforms, #22000, but I don't want to try the migration to split the platform and execution platform concepts yet.

Adding @gregestren as reviewer, but I am always interested in comments and suggestions.

[katre]

First question: I'm not wild about the name enabled_settings, but couldn't come up with anything better. Both exec_settings and target_settings invite confusion because there is only one configuration, but any settings that check constraints will check the target platform for that configuration.

Any suggestions on naming would be very appreciated!

[gregestren]

What confusion do you mean? e.g. what does target_settings make ambiguous?

[katre]

It's the same question that @trybka asked: does this apply to using a platform as a target or an exec? Does this compare the settings against the target configuration or the exec configuration? Why is it called target_settings if it's deciding whether a platform is an exec platform?

[gregestren]

required_settings? Just use a select() on constraint_values that can optionally set some "nothing matches" constraint?

I could see enable_settings being ambiguous about whether it's a condition to check or something it's actually setting. I like the idea of something more directly communicating "this is what I need to activate".

[katre]

I think required_settings makes sense, and making sure to document that it only applies to using the platform for execution.

Thanks!

[trybka]

Does this apply to target platforms (specified via --platforms=)?
That is, would this allow one to restrict targeting a platform with a setting?

[katre]

No, this would only apply to execution platforms. This is probably the biggest problem with the name enabled_settings.

Technically, using this to gate target platforms would be tricky: the target platform is part of the configuration, and we need a configuration in order to evaluate config_setting targets, so needing to evaluate config_setting to check the target platform is circular. I think it's just barely possible (a failure would lead to an immediate error, not a fallback), but I don't want to deal with it at this point.

[gregestren]

. Once the flag is set, the enabled_settings will match, and the remote_extra_logging platform will be available (and is registered before the other).

Doesn't this depend on the order of registration? Is that what "is registered before the other" means?

[katre]

That's correct: if all execution platforms are enabled (either by not specifying enabled_settings, or because all of the settings match the current configuration) then the registration order is used to pick a preferred execution strategy, the same as today.

[gregestren]

This makes the argument for separating out exec resolution logic into something else stronger.

I think of platforms as a bag of properties for a machine you want your build to interact with. It's a nice abstraction to keep: simple, straightforward, universal. How we integrate with platforms is its own logic. It's weird to add API support to that effect that only triggers under certain scenarios that are technically separate from the platform definition itself.

That creates ambiguities like what Tom asked above. Similarly, if someone uses the same platform for target binaries and build execution, is it weird if it has enabled_settings and you set --platforms=foo and have to remember those settings don't kick in there?

Thinking through #22000 more, could it make sense to have exec_platform be a wrapper over a platform similar to how toolchain wraps toolchain implementations? Like

  platform(name = a)
  platform(name = b)
  exec_platform(
    platform = ":b",
    enabled_settings = [...]
  )

It's yet another abstraction. But that logic has to live somewhere.

I realize you're not trying to address that just yet. To be constructive here I'd love to understand what our end API goal is. My current assertion is that resolution logic principally belongs somewhere else, and we keep platforms as just a bag of properties. Any holes there? Do platform-based flags already disrupt that idea?

[katre]

I like the idea of a wrapper for a platform to make it useful as an exec platform (this also addresses worries about the autodetected host platform: it's the host platform, and the default for the target platform, and it's usable as an exec platform, which is a lot to unpack if we split these out).

I'm not sure what the end state for the API is. A quick dump from a half-done doc I wrote:

Target platforms have:

• An optional parent to inherit from
• Constraint values
• Key-value settings?
• A short name to use for output paths
  • Is this also a key-value setting? A direct attribute? Unknown

Exec platforms have:

• An optional parent to inherit from
• Constraint values
• Definitely key-value settings
  • Also the legacy remote_execution_properties that we haven't removed yet
• A way to selectively enable/disable based on configuration (this proposal)
• A list of allowed execution strategies (see Execution Platforms vs Strategies)

Also, every exec platform can be a target platform (after the execution transition), which points towards your wrapper idea.

It's a lot, and I'd like to spend some time to tease out the different parts. Can we move forward on this proposal and acknowledge that whatever we do, the migration to split target and exec platforms will be complicated?

[gregestren]

LGTM aside from naming thoughts I mentioned in the other comment. Given my feedback, choose whatever makes most sense to you on the attribute naming.

[sluongng]

This is a sensible proposal that enables exec platform selection to be more flexible, thus letting users build with more variety of exec platforms more easily. We previously saw that folks would rather modify the exec_properties on the target-level to override the exec platform's properties so that they would not have to deal with multiple platforms.

I think the downside of this approach is that: this is yet, another migration. Currently, in the Open Source Bazel ecosystem, we see that folks are still migrating off --cpu / crosstools setup into the new platform toolchain resolution setup. With this transition, there is a lot of pain and confusion, especially around how the automatic resolution works: what's available, what's being matched against what, etc... I raised #22240 to request surfacing this information via BEP so that we could build better tools to help explain this information to Bazel users.

I suspect we would also need similar information with regarding to Configuration if this proposal is accepted. Right now the Configuration event is insufficient to help explain such automatic resolution as no related build settings are being surfaced.

bazel/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto

Lines 443 to 452 in 2a61fe7

	// Payload of an event reporting details of a given configuration.
	message Configuration {
	string mnemonic = 1;
	string platform_name = 2;
	string cpu = 3;
	map<string, string> make_variable = 4
	;
	// Whether this configuration is used for building tools.
	bool is_tool = 5;
	}

[katre]

the downside of this approach is that: this is yet, another migration

I'm curious what you see as being a migration, since I'm not proposing work on a new exec_platform rule in any form. This proposal adds a new attribute to the existing platform rule, and if it is not used, the previous behavior will remain: the platform will be available as an execution platform if registered.

I agree with you that we need better discoverability around platforms and toolchains, and think #22240 is headed in the right direction, but I consider it to be separate from this work (and, frankly, I don't have enough insight into BEP to be able to help design it). I'm definitely available to discuss what data is available that could be added to BEP if the users can agree on the most useful format.

I don't consider #22240 to be blocking for this proposal: do you disagree?

[sluongng]

Ah I was thinking of the exec_platform suggestion above, which gona create a new way to define exec platform on top of the existing way, and that probably would require a code migration to clean up the old way some sort? Without exec_platform, this proposal would not need a code migration as things should still work as-is.

I don't think #22240 technically blocks the proposal, but in a sense, a barebone of this proposal implementation will add to the usability problem that #22240 is aiming to solve.

---

Taking a step back, here is my perception of the current OSS Bazel user personas:

1. The developer: they don't know much about Bazel, all they know about are the targets being linked to each other via BUILD file declaration.

2. The build team: these folks are familiar with Bazel, some have authored their own rules, others have tried setting up RBE. These folks often understand Targets + Rules -> Actions, and Actions are sent over to RBE services with Platforms and Toolchains, but usually their understanding of Platforms and Toolchains is lacking. Most of these folks don't know/care about Configuration.

3. OSS rules author, Bazel contributors: better understanding of Platforms and Toolchains since they have to write and maintain these definitions. An improved understanding of Configuration vs persona (2) and spent a lot of time inspecting/debugging to get things right.

So without better API and toolings to surface these Platforms, Toolchains, and Configurations concepts, personas (1) and (2) gona suffer a lot.

With that said, I would recommend putting the --toolchain_resolution_debug information into BEP and to implement something similar for this proposal to help with exec platform resolution debug.

[katre]

I agree with you that all of this would be useful: I just don't think this proposal is the right place to tackle figuring out what should be added to the BEP.

[sluongng]

I agree with you that we need better discoverability around platforms and toolchains, and think #22240 is headed in the right direction, but I consider it to be separate from this work (and, frankly, I don't have enough insight into BEP to be able to help design it). I'm definitely available to discuss what data is available that could be added to BEP if the users can agree on the most useful format.

@katre I sent #23019 to kick-start the conversation. Looking forward to your feedback on this.