Install logic switched to ele-testhelpers #191

thehejik · 2024-10-24T16:06:15Z

Fixes #189

Totally work in progress - the suite and install files were borrowed from turtles-e2e - needs to be adapted for HP.

RANCHER_PASSWORD=mypwd \
RANCHER_HOSTNAME=1.2.3.4.nip.io \
RANCHER_VERSION=latest/2.9.2 \
INSTALL_K3S_VERSION=v1.28.10+k3s1 \
make prepare-rancher

Todo:

create a normal user??? - NOT USED
install helm - already in Makefile as install-helm target - now always installs latest
install nightly operators - ~~so far kept in Makefile~~ moved to install_suite.go
waiting for rancher-webhook pods already implemented, fixes Flake: Fix rancher-webhook missing for test cases #183
proxy - run squid in docker, config k3s, certmanager and rancher
process squid logs
combine nighly x proxy x normal - no big change - only that all scenarios are driven by env variables NIGHTLY_CHART and RANCHER_BEHIND_PROXY (use when != "") and calling make prepare-rancher
code cleanup

Signed-off-by: Tomas Hehejik <[email protected]>

Signed-off-by: Parthvi Vala <[email protected]>

…rt for v2.10-head Signed-off-by: Tomas Hehejik <[email protected]>

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik · 2024-11-11T19:31:12Z

Notes for initial proxy support:

add support for proxy in k3s, cert-manager and rancher via command:

RANCHER_BEHIND_PROXY=anything \
RANCHER_PASSWORD=mypasswd \
RANCHER_HOSTNAME=1.2.3.4.nip.io \
RANCHER_VERSION=latest/devel/2.9 \
INSTALL_K3S_VERSION=v1.28.10+k3s1 \
make prepare-rancher

proxyUrl is hardcoded
Workflows not modified
Cleanup required

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik · 2024-11-13T10:59:41Z

required PR rancher-sandbox/ele-testhelpers#67

* initial cleanup * workflows not adapted yet * testhelpers not merged/bumped Signed-off-by: Tomas Hehejik <[email protected]>

thehejik · 2024-11-13T12:40:36Z

Notes for previous commit

function complete
nightly charts supported (needs change in testhelpers)
workflows not adapted yet
initial cleanup

Used command to deploy rancher with proxy and with nightly hosted providers chart:

PROVIDER=aks \
NIGHTLY=yes \
RANCHER_BEHIND_PROXY=yes \
RANCHER_PASSWORD=mypassword \
RANCHER_HOSTNAME=1.2.3.4.nip.io \
RANCHER_VERSION=latest/devel/2.10 \
INSTALL_K3S_VERSION=v1.30.6+k3s1 \
make prepare-e2e-ci-rancher-hosted-nightly-chart-behind-proxy

Verified values helm get values rancher -n cattle-system -o yaml:

bootstrapPassword: mypassword
extraEnv:
- name: CATTLE_SERVER_URL
  value: https://1.2.3.4.nip.io
- name: CATTLE_AGENT_IMAGE
  value: rancher/rancher-agent:v2.10-head
- name: CATTLE_SKIP_HOSTED_CLUSTER_CHART_INSTALLATION
  value: "true"
hostname: 1.2.3.4.nip.io
noProxy: 127.0.0.0/8,10.0.0.0/8,cattle-system.svc,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
proxy: http://172.17.0.1:3128
rancherImageTag: v2.10-head
replicas: 1
useBundledSystemChart: true

* ele-testhelpers bumped to support extraFlags for rancher * main.yaml adapted * proxyHost not hardcoded anymore but using fallback when no set * Makefile cleanup and env vat check added for rancher installation * still contains debug info Signed-off-by: Tomas Hehejik <[email protected]>

thehejik · 2024-11-14T16:10:23Z

First testrun, enabled proxy and nightly on AKS with p0_provisioning:
https://github.com/rancher/hosted-providers-e2e/actions/runs/11841053192

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik · 2024-11-14T16:25:42Z

After fix for installing nightly_charts:
https://github.com/rancher/hosted-providers-e2e/actions/runs/11841322069

valaparthvi

Perhaps hosted/preparation can be moved out of hosted and into the root directory since it is not directly related to testing hosted providers.
On main.yaml L#341 Collect logs step, can you modify it to run if steps.prepare-cluster.outcome == 'success', even if the job is canceled? I recently came across a case where I need to cancel a long running job, but I still wanted to see the logs from the tests that ran.

Overall great work on this, thank you! 🚀

valaparthvi · 2024-11-15T12:16:04Z

hosted/preparation/squid.conf

@@ -0,0 +1,82 @@
+#


There is a squid.conf in .github/scripts. Can you verify the contents of this file with it and delete it?

I made a copy of the original file in hosted/preparation/squid.conf because the original location was harder to access , after moving the code to root dir I can access it easily again. I reverted back to original location.

valaparthvi · 2024-11-15T12:17:45Z

hosted/preparation/suite_test.go

+	if rancherHostname == "" {
+		Fail("RANCHER_HOSTNAME environment variable is required")
+	}


nit:

Suggested change

if rancherHostname == "" {

Fail("RANCHER_HOSTNAME environment variable is required")

}

Expect(rancherHostname).ToNot(BeEmpty(), "RANCHER_HOSTNAME environment variable is required")

}

valaparthvi · 2024-11-15T13:13:21Z

hosted/preparation/suite_test.go

+	if rancherHostname == "" {
+		Fail("RANCHER_HOSTNAME environment variable is required")
+	}
+	rancherVersion = os.Getenv("RANCHER_VERSION")


REQUIRED_VARS := INSTALL_K3S_VERSION RANCHER_HOSTNAME RANCHER_PASSWORD RANCHER_VERSION

rancherVersion is a required env var, we should add a Expect(<field>).ToNot(BeEmpty()) check same for the other fields as well. WDYT?

valaparthvi · 2024-11-15T13:20:53Z

hosted/preparation/suite_test.go

+	// Extract Rancher Manager channel/version to install
+	if rancherVersion != "" {
+		// Split rancherVersion and reset it
+		s := strings.Split(rancherVersion, "/")
+		rancherVersion = ""
+
+		// Get needed informations
+		rancherChannel = s[0]
+		if len(s) > 1 {
+			rancherVersion = s[1]
+		}
+		if len(s) > 2 {
+			rancherHeadVersion = s[2]
+		}
+	}


If RANCHER_VERSION is a required field, and assuming we have already checked that it is not empty, we do not need to recheck here.

Suggested change

// Extract Rancher Manager channel/version to install

if rancherVersion != "" {

// Split rancherVersion and reset it

s := strings.Split(rancherVersion, "/")

rancherVersion = ""

// Get needed informations

rancherChannel = s[0]

if len(s) > 1 {

rancherVersion = s[1]

}

if len(s) > 2 {

rancherHeadVersion = s[2]

}

}

// Extract Rancher Manager channel/version to install

// Split rancherVersion and reset it

s := strings.Split(rancherVersion, "/")

Expect(len(s)).To(BeNumerically(">=", 2))

rancherVersion = ""

// Get needed information

rancherChannel = s[0]

rancherVersion = s[1]

if len(s) > 2 {

rancherHeadVersion = s[2]

}

latest/latest, latest/2.9.3[-rc2], prime/2.9.3, prime/devel/2.9, prime-optimus/2.9.3-rc2

Also, if it is provided, on splitting the string, its length would be >= 2, you would need to have the channel and version.
prime/devel/2.9 -> version=devel headVersion=2.9, prime-optimus/2.9.3-rc2 -> version=2.9.3-rc2

valaparthvi · 2024-11-15T13:27:52Z

.github/workflows/main.yaml

-            fi
-          elif [ ${{ inputs.proxy }} == true ]; then
-            make prepare-e2e-ci-rancher-behind-proxy
+            export NIGHTLY_CHART="enabled"


Can we use a boolean value for this instead of a string? Same for proxy env.

For nightly it would be possible to use boolean but it's problematic for proxy because testhelpers function for installing Rancher accepts string anyway when requesting proxy. I would rather keep both as strings for now.

Also I removed the else part as it is not actually needed.

valaparthvi · 2024-11-15T13:36:03Z

hosted/preparation/install_test.go

+
+		By("Installing K3s", func() {
+			// Get K3s installation script
+			fileName := "k3s-install.sh"


Can you add defer to delete this file once the test finishes?

valaparthvi · 2024-11-15T13:40:23Z

hosted/preparation/install_test.go

+			installCmd := exec.Command("sh", fileName)
+			// INSTALL_K3S_VERSION should be already set by the action or user's shell
+			installCmd.Env = append(os.Environ(), "INSTALL_K3S_EXEC=--disable metrics-server --write-kubeconfig-mode 644")


Do you think there is a way to install k3s without downloading the file?

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$INSTALL_K3S_VERSION sh -s - server --cluster-init --tls-san=$RANCHER_HOSTNAME

I usually use the above command, it installs k3s without downloading the script as far as I know.

valaparthvi · 2024-11-15T13:45:13Z

hosted/preparation/install_test.go

+				GinkgoWriter.Printf("K3s installation loop %d:\n%s\n", count, out)
+				count++
+				return err
+			}, tools.SetTimeout(2*time.Minute), 5*time.Second).Should(BeNil())


nit:
Add a failure message to Eventually. Also add it to other eventually blocks if you go ahead with the suggestion.

Suggested change

}, tools.SetTimeout(2*time.Minute), 5*time.Second).Should(BeNil())

}, tools.SetTimeout(2*time.Minute), 5*time.Second).Should(BeNil(), "Timed out while waiting for k3s to install")

AFAIK Should is not designed to return a custom message on failure :(

It does. We are using it at a couple of places within our repo. Search for "Timed out", you'll find it. Let me know if I am mistaken 😓 .

valaparthvi · 2024-11-15T13:50:09Z

hosted/preparation/install_test.go

+
+		By("Waiting for K3s to be started", func() {
+			// Wait for all pods to be started
+			checkList := [][]string{


I think somewhere around here you need to ensure a KUBECONFIG env var is exported. When I tried running this locally without a KUBECONFIG exported, it errored out.

Perhaps, you can add KUBECONFIG to the REQUIRED_VARS list and also ensure it is .ToNot(BeEmpty()) in the BeforeSuite?

Signed-off-by: Tomas Hehejik <[email protected]>

WIP Install logic switched to ele-testhelpers

036e4ba

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik marked this pull request as draft October 24, 2024 16:06

thehejik self-assigned this Oct 24, 2024

thehejik added 5 commits October 25, 2024 11:29

WIP first try in CI - w/ nightly; w/o proxy

76422ba

Signed-off-by: Tomas Hehejik <[email protected]>

WIP fix for first try in CI - setup-go moved; typo

e7add14

Signed-off-by: Tomas Hehejik <[email protected]>

WIP Make test steps dependent on prepare-cluster step; install helm

9ae410b

Signed-off-by: Tomas Hehejik <[email protected]>

WIP do not set server-url by helper; use recent squid image

bab5046

Signed-off-by: Tomas Hehejik <[email protected]>

WIP stdout visible for helm

fcf0056

Signed-off-by: Tomas Hehejik <[email protected]>

This was referenced Oct 31, 2024

Wait for fleet-controller and rancher-webhook deployments #185

Closed

Use stgregistry for rancher v2.7-head and 2.8-head #150

Closed

valaparthvi and others added 3 commits November 4, 2024 11:58

Update eks.yaml

2138376

Signed-off-by: Parthvi Vala <[email protected]>

WIP Bump testhelpers to reflect new v2.9-head location and also suppo…

6cb28d6

…rt for v2.10-head Signed-off-by: Tomas Hehejik <[email protected]>

WIP Initial support for proxy via RANCHER_BEHIND_PROXY env

4d39698

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik added 2 commits November 12, 2024 11:47

WIP run proxy in separate By block, wait for capi-controller-manager

49de1f2

Signed-off-by: Tomas Hehejik <[email protected]>

WIP bump testhelpers

24e9e71

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik force-pushed the testhelpers branch from 3619e3e to 24e9e71 Compare November 12, 2024 10:50

WIP function complete \

20ffb9d

* initial cleanup * workflows not adapted yet * testhelpers not merged/bumped Signed-off-by: Tomas Hehejik <[email protected]>

Fix by adding export NIGHTLY_CHART in action

79653e9

Signed-off-by: Tomas Hehejik <[email protected]>

thehejik changed the title ~~WIP Install logic switched to ele-testhelpers~~ Install logic switched to ele-testhelpers Nov 15, 2024

thehejik marked this pull request as ready for review November 15, 2024 10:39

valaparthvi requested changes Nov 15, 2024

View reviewed changes

thehejik added 3 commits November 15, 2024 15:55

Move rancher installation code to root

5c00412

Signed-off-by: Tomas Hehejik <[email protected]>

Always collect logs if the local cluster is ready

d3d5291

Signed-off-by: Tomas Hehejik <[email protected]>

Better env check (both Makefile and go), rancherVersion parsing improved

50ffc6b

Signed-off-by: Tomas Hehejik <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install logic switched to ele-testhelpers #191

Install logic switched to ele-testhelpers #191

thehejik commented Oct 24, 2024 •

edited

Loading

thehejik commented Nov 11, 2024 •

edited

Loading

thehejik commented Nov 13, 2024

thehejik commented Nov 13, 2024 •

edited

Loading

thehejik commented Nov 14, 2024

thehejik commented Nov 14, 2024

valaparthvi left a comment

valaparthvi Nov 15, 2024

thehejik Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

thehejik Nov 15, 2024

thehejik Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

thehejik Nov 15, 2024

valaparthvi Nov 15, 2024

valaparthvi Nov 15, 2024

	}, tools.SetTimeout(2time.Minute), 5time.Second).Should(BeNil())
	}, tools.SetTimeout(2time.Minute), 5time.Second).Should(BeNil(), "Timed out while waiting for k3s to install")

Install logic switched to ele-testhelpers #191

Are you sure you want to change the base?

Install logic switched to ele-testhelpers #191

Conversation

thehejik commented Oct 24, 2024 • edited Loading

thehejik commented Nov 11, 2024 • edited Loading

thehejik commented Nov 13, 2024

thehejik commented Nov 13, 2024 • edited Loading

thehejik commented Nov 14, 2024

thehejik commented Nov 14, 2024

valaparthvi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehejik commented Oct 24, 2024 •

edited

Loading

thehejik commented Nov 11, 2024 •

edited

Loading

thehejik commented Nov 13, 2024 •

edited

Loading