Consider adding runtime, windowsdesktop, and aspnetcore to build table

[mmitche]

Add the installers for runtime, windowsdesktop, and aspnetcore to the build table, at least for 5.0+

[marcpopMSFT]

@wli3 for visibility as he has a tool to simplify updating that table.

[dagood]

Why in this repo, rather than the repos where these installers are built? (There's also some tooling in dotnet/runtime and dotnet/windowsdesktop for readme tables.)

[wli3]

I refer not to add more complexity to this table. It is still a mess to update. Ideally it should just be a static website.

[dougbu]

@davidfowl said in an email about the hosting bundles…

No, it should be with the other installers. ASP.NET Core is included in .NET when you click on the current installer and this should be there as well.

[davidfowl]

No it needs to be in sync with whatever build was ingested into the installer here. I don't see why this is a problem. This repository is the aggregate of all the other repos so it should map to the ci version of the official downloads page.

[mmitche]

Today that is not possible. We know the info, but we do not generate links for the individual components within the installer. So we'd need to do some work to do this.

Also, you sure that's what you want? Or do you want the latest of each individual repo? The installer sometimes lags significantly behind the latest of the other repos.

[mmitche]

Today that is not possible. We know the info, but we do not generate links for the individual components within the installer. So we'd need to do some work to do this.

Also, you sure that's what you want? Or do you want the latest of each individual repo? The installer sometimes lags significantly behind the latest of the other repos.

Also of note is that with .NET 6's workload based installer, it probably makes more sense just to link latest from each repo, because as far as I understand, the installer repo will not package up individual runtime repos (aside from the core runtime required to actually run the sdk).

[davidfowl]

Let's start over then. Pretend I'm a customer (this happened in real life) and I'm looking for the ci build equivalent of this page https://dotnet.microsoft.com/download/dotnet/5.0. Am I now forced to go to all the repos and find out which installers map to the versions here?

Ideally this landing page would be a sufficient replacement. If not, we should make a new place

[wli3]

For CI builds multiple SDK version could map to single aspnet version (say runtime keeps changing).

[mmitche]

Let's start over then. Pretend I'm a customer (this happened in real life) and I'm looking for the ci build equivalent of this page https://dotnet.microsoft.com/download/dotnet/5.0. Am I now forced to go to all the repos and find out which installers map to the versions here?

Ideally this landing page would be a sufficient replacement. If not, we should make a new place

I think we still need to be a little clearer on what the customer wants. What are they intending to use the links for? There are 3 possibilities that I see:

1. We match exactly what is on dot.net - This means coherent builds. The inherent problem with this is that we reach coherent builds in development branches extremely rarely. This hypothetical page would basically update only as often as we ship previews. This is not for failings in the repos or problems with dependency flow, but largely because dependencies flow at different rates through the graph. And because the runtime repos actually get redistributed in interesting ways in other runtime repos, there are actual cases where binaries from more than one runtime build would show up in the installer.
2. We sort of match what is on dot.net - Let's say we put the latest SDK, and then runtime, windows desktop, and aspnetcore binaries that inserted directly into that SDK. At this point, we're sort of ignoring the coherency issue and saying the versions that inserted into the SDK are what "matters most", even if some binaries from, say, dotnet/runtime, got repackaged along the way in aspnetcore. The issues I see with this are that:
   • Blockage in the SDK dependency flow could mean that some links go a while without updates.
   • The workload based installer may mean that we eventually don't even have a real insertion of aspnetcore and windowdesktop. The "insertion" either doesn't happen at all (no PR to SDK), or doesn't have any effect on the produced outputs in the repo,
3. We publish the latest of each repo - We just place the latest of each runtime on a unified page. That installer may not have ingested the latest runtime or aspnetcore, but that's probably fine, it will most likely still work great if installed. It's also closer to a workload model that we're going for in .NET 6.

[davidfowl]

I don't think coherence is a problem. We have repositories ingesting bits from other repositories. That's how you get the "right" version. This repository aggregates other repos so AFAIK it just needs to point to whatever it ingested. I'm not sure why this is complicated...

[mmitche]

I'm not sure why this is complicated...

It's complicated because there is a ton of subtlety that is hidden away when we have a coherent build. Implicitly installer is ingesting multiple versions of the runtime when the product is incoherent. And those multiple versions do end up in the installer tarball, mostly as files within the other shared frameworks.

This repository aggregates other repos so AFAIK it just needs to point to whatever it ingested.

We should be able to do this. I will need to follow up with the installer folks though, w.r.t. to the selectable workload installer. I am not positive whether there will be an aggregator after that is implemented.

[mmitche]

I don't think coherence is a problem

One more thing. Coherence may be a problem depending on what the expectations are for someone using this site. When they use dot.net, the expectation is that things work. If we simply link what installer ingests, there is no guarantee that aspnetcore even works at all. For instance, a breaking change was checked into the RC1 branch at the end of the cycle. This change required reaction in aspnetcore, but not in windowsdesktop or other repos. So this change flowed into installer, meaning that the version of aspnetcore in the installer would just simply fail to run on top of the version of the runtime present in the installer.

Do we care about these scenarios? Or is it really just "use at your own risk, may not work at all"?

[davidfowl]

@mmitche How is that different from what happens right now?

[mmitche]

From the context of someone using the installer, they would see the same behavior, So if this is really just about making explicit the links of what is included in the current installer, with no expectation of how recent the build of runtime, aspnetcore, windowsdesktop, etc. will be, then just creating a site with the current included builds of other repos sounds good to me.

We don't have a generic way to generate these links today. The current aka.ms link generation is generic, but it basically just strips away version numbers out of blobs and adds an aka.ms prefix. There's no automated way for that to happen for included components (because there's no generic way to identify what is an included component).

Some options I can see:

• Add a new feature to publishing which would allow for "included components" to be identified in the manifest. Then auto-generate links for those included components. We are busy rolling out some publishing changes for .NET 5 so this would take a while to implement.
• Have installer auto-generate a page with the links on each build, and publish this like any other artifact, and this page would be stable linked with aka.ms like the rest of the installer artifacts. This is probably a quicker, though less generic, implementation.

@wli3 What do you think?

[wli3]

Both sounds reasonable. It is generally a good thing. However, for SDK team, I still think this task has low priority. I'd like to have more data on how many customer consume daily build of SDK and how many people need to get the pieces individually. From my impression, there are very few people. And according to the net5 and net6 schedule, we are unlikely to work on it any time soon.

And I still prefer SDK team not owning this feature. The central engineering team should have better access like darc and aka.ms/bar to build it. And they are better situated for cross cutting and live site like concerns.

[mmitche]

And I still prefer SDK team not owning this feature. The central engineering team should have better access like darc and aka.ms/bar to build it. And they are better situated for cross cutting and live site like concerns.

Yeah, agreed. I'd like to add a feature which would allow for identification of redistribution, which would then allow us to auto-generate links. This would still require some work on the SDK teams side (to identify which versions are being repackaged), but it would be minor.

We can't generate such links solely off of the info we have today (just version.details.xml) because after we ship, incremental servicing (e.g. around the ref packs), will mean that multiple versions of the runtime, aspnetcore, etc. will be referenced within the repo. And Maestro has no way of interpreting which one is important.

[davidfowl]

How are the links generated today?

[mmitche]

How are the links generated today?
Any repo may publish various blobs (e.g. Sdk/5.0.100-rc.1.1234.5/dotnet-sdk-5.0.100.rc.1.1234.5-win-x64.zip). These are published to a location based on what channel the build is assigned to (e.g. a public build will go to dotnetcli.blob.core.windows.net/dotnet).

To generate a stable link, we need to remove the parts of that link that will change on a build to build basis. Basically the version numbers. So the infra will identify, by heuristic, meaningful version numbers in that path, remove them, and then prepends information about the channel as part of an aka.ms link. So for instance, for rc1, that link would become:

aka.ms/dotnet/net5/rc1/Sdk/dotnet-sdk-win-x64.zip

This is a stable link that can be linked on a page, which is what we link on the installer page right now. The same sort of links are generated for aspnetcore, runtime, windowsdesktop, etc. components.

Now, let's get to how we identify components. Let's say you want to identify the runtime that was inserted into installer. You want a stable link again, but you can't use aka.ms/dotnet/net5/rc1/Runtime/dotnet-runtime-win-x64.zip. This would be the latest build of runtime, not the build that was inserted into the installer. We know the various versions of the runtime that are inserted into the installer via the version details files, but we don't know how installer actually interprets them. Installer (or any other repo for that matter) itself needs to identify what blobs/versions are redistributed in a meaningful way (e.g. the targeting pack is less meaningful than the runtime that aspnetcore will run on). If we have that info, we could use one of the two options above:
#8315 (comment) to generate either a page, or a new set of information that could be handled generically by the infra.