[BUG] 3006.9 master not able to target minion by grain correctly

[tjyang]

Description
For example, salt -G "os:macos" test.ping will display correct output of macos minions but right after the correct display it also display other minions and make them as not responded in time(salt-run jobs.lookup_jid xxxxx).
I believe this is a regression from early version of salt.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• [ x] on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• [ x] onedir packaging
• [ x] used bootstrap to install

Steps to Reproduce the behavior
Ex: two real macos minions responded but it also match the other 300 non-macos minions.

[me@salt01]$ sudo salt -G "os:macos" test.ping
<snipped>
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 300
# of minions returned: 2
# of minions that did not return: 300
# of minions with errors: 0
-------------------------------------------
ERROR: Minions returned with non-zero exit code
[me@salt01]$

• this is rocklinux case: 200 non rocky minion also got contacted for ping result and failed.

[me@salt01 ]$ sudo salt -C 'G@os:rocky' test.ping
<snipped>
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 400
# of minions returned: 200
# of minions that did not return: 200
# of minions with errors: 0
-------------------------------------------
ERROR: Minions returned with non-zero exit code
[me@salt01 ]$ 

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.16.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Additional context

• Reference:
  • [BUG] salt version 3006.8 upgrade to 3006.9 grains issue #66774
  • [BUG] Minion did not return. [No response] with some random minions #57750

[ewaldmire]

FYI: #66973

[tjyang]

FYI: #66973
Thanks for the link, but it is about removing macos build infra server. not about salt-master's issue using os grain to poke minions.

[tjyang]

• Issue 1: Minion need to be restarted.

salt -G 'os:redhat' test.ping # should returns minons that are redhat

Workaround solution by restarting unnecessary targeted minions .

Restarting the windows minions.
after the salt-master upgrade cleared the odd unnecessary targeting we were seeing.

• Issue 2: If a host is down at time of running grain targetting the salt -G 'os:redhat' test.ping will failed and showing all those disconnected minion name.