OPA: Reduce impact of filter creation time on routes

[mjungsbluth]

Is your feature request related to a problem? Please describe.
The Open Policy Agent filters wait for a default of up to 30s to get functional, filter creation for example fails if the bundle cannot be loaded in time or the control plane for bundle / discovery download is unavailable / rejects the request.

In the current there are multiple issues this can cause:

• OPA instances are shared amongst filter instances and cached but created serially. In case routes use a lot of different bundles, this can slow down the creation (due to a fairly broad mutex)
• Even if the discovery and bundle plugin have already reported errors, the way OPA reports plugin success, creation waits until timeout (could be fail fast)
• Blocking filter creation too long can block Endpoint updates, i.e. can lead to intermittent 502s on the current routes

Describe the solution you would like

For the startup, the the implementation can be changed to fail fast if errors in discovery / bundle download are detected. The consequence of this would be that the filter would also fail fast even on intermittent 5xx in bundle download or network timeouts.

OPA instance creation can be parallelized (i.e. the current mutex on the registry could just guard creation of a sync.Once per instance and the actually startup of the instances can run in parallel for multiple filters using different bundles.

We can optionally make the creation of the filter asynchronous in the sense that bootstrap is done in the context of the filter but there is no wait on successfully loading the bundle / finishing discovery. The consequences of this would be that route creation is very fast but potential problems would be pushed to the data plane, i.e. requests would either block or return 5xx status for a while. The latter would be especially problematic if you introduce the filter into an existing route.

Would you like to work on it?
Yes