Bootstrapping {mini}conda from scratch

[mingwandroid]

From: conda-forge/conda-forge-pinning-feedstock#1187

During the process of bootstrapping packages to a new architecture, it is sometimes necessary to build packages in bootstrapping mode. For example, when the full functionality of the package isn't available yet, but the package is necessary to build other packages will make the full functionality available later (in a regular non-bootstrapping) build.

In order to decide whether or not a build is run in bootstrapping mode, I suggest adding a new variable to conda_build_config.yaml named bootstrapping whose value is either "yes" or "no".

In a recipe, we can then distinguish packages created by using:

build:
{% if bootstrapping == 'yes' %}
  number: 0
  string: h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}_bootstrapping
{% else %}
  number: 1
{% endif %}

For example, we can disable build requirements the following way:

requirements:
  build:
{% if bootstrapping != 'yes' %}
    - libtool  # [unix]
    - bash     # [unix]
{% endif %}

In some cases it may the necessary to have access to the bootstrapping variable from within build.sh:

  script_env:
    - BOOTSTRAPPING={{ bootstrapping }}

[mingwandroid]

Ping @ilanschnell, @nehaljwani, @leej3, @jakirkham, @isuruf

[isuruf]

I'm in favour of adding the variable bootstrapping as it will have other uses to break cycles in new python versions, abi pins, etc.

However, I'm not sure if we want to support it for the dependencies of the ctng-compilers. ctng-compilers feedstock already supports building a cross compiler (build==host!=target). This can be used for building a cross-native compiler (build!=host==target), but it isn't supported yet. If this can be achieved, then we can bootstrap a platform with ease.

Note that we bootstrapped osx-arm64 platform from scratch with cross compiling after fixing a few recipes.

[isuruf]

To generalize this, it's okay to have cycles in build and host, but it's not okay to have have cycles within host.

[mingwandroid]

However, I'm not sure if we want to support it for the dependencies of the ctng-compilers. ctng-compilers feedstock already supports building a cross compiler (build==host!=target). This can be used for building a cross-native compiler (build!=host==target), but it isn't supported yet. If this can be achieved, then we can bootstrap a platform with ease.

There are only a handful of these dependencies (about 10 packages). We do not have time to investigate cross scenarios here and I believe supporting bootstrapping natively as well as via cross is a good thing.

[katietz]

To generalize this, it's okay to have cycles in build and host, but it's not okay to have have cycles within host.

I agree to this statement, nevertheless we have especially cycles also about run requirements, and especially about testing. A common package illustrating the test scenario isssues is 'patch' feedstock. One of the first cyclic dependencies we can see in autotools, which are in general just 'build:' section related. Another interesting usage are tools, which need additional infrastructure of other languages, like 'ripgrep', which depends on rust, or ctng-compilers depending on cmake, which is actually already a pretty complex package used here for fortran tests.
A sample of host/run-req cyclic dependency is the 'chardet' package.

[katietz]

That cross-compilers can be used to do bootstrapping is right, but overcomplicate bootstrap process to a different target without much use. A cross-compiler (especially one for a new architecture) contains likely setup/configure/env bugs, and is by default not fully tested for given architecture, as remote-testsuite run is not a standard feature of common bootstrap of a cross-compiler. So in general it is to be preferred to use actual native system-compiler, and do a full 3 stage bootstrap with its test-runs, which detects early configure/code-gen/etc issues related to the compiler.
So I think supporting cross is fine, but for a sane bootstrap I would avoid it.

[isuruf]

Sure, but how would you bootstrap conda-build without conda-build? pip install conda-build?

cross-compiler (especially one for a new architecture) contains likely setup/configure/env bugs, and is by default not fully tested for given architecture, as remote-testsuite run is not a standard feature of common bootstrap of a cross-compiler.

I agree with you that fully testing is needed. What I'm suggesting is that we build the cross compiler, build the dependencies of the native compiler, and start from scratch with the native compiler. We anyways have to go twice.

but overcomplicate bootstrap process to a different target without much use.

The use will be the ease of porting to a new architecture. Otherwise, somebody will have to make sure that pip install conda-build with all its dependencies like lief, etc are built without the use conda-build and go forward.

[isuruf]

I still consider osx-arm64 platform as experimental because it was cross-compiled and not tested. Once we get native CI running, we'll start again with the compilers and rebuild the stack and run tests.

[katietz]

well, I would use a helper script 'conda-from-scratch', which can be build by using the system-compiler itself. By this you won't need to use any cross for that process.

[katietz]

I still consider osx-arm64 platform as experimental because it was cross-compiled and not tested. Once we get native CI running, we'll start again with the compilers and rebuild the stack and run tests.

Strongly agreed