Improvements to internal image unpacking abstractions

[joelanford]

For now, I'm opening this issue to collect ideas for ways to improve our image unpacking, caching, and storage abstractions, for both catalogd and operator-controller.

The ideas I can think of now are:

1. Async image pulling and unpacking. If an image needs to be unpacked, spawn a separate goroutine to do it and signal the reconciler when the unpack completes (or fails).
2. Combine the catalogd and operator-controller implementations. They are similar but not binary compatible. We likely need a more generic "request" type that can be used for both ClusterCatalogs and ClusterExtensions.
3. We need a way to ask: "what is the current state of $id" in a way that we can always reconstruct status. Of note for catalogd is the extra wrinkle for images of wanting to skip all registry interactions, even for tag-based references, based on a configured poll interval. We can only skip unpacking if we know that we are actually already serving the expected content, and we may need to update status even if we aren't unpacking.
4. Could we simplify or remove catalogd's storage implementation by giving more responsibility to the Unpacker, where someone using the Unpacker can provide a function that manifests an image onto the filesystem in whatever way they want. For example:
   • Catalogd could walk the FBC root within the image filesystem and directly write all.json as part of the unpack process, thereby eliminating the extra unpack cache, disk I/O, etc.
   • Operator-controller could extract just the bundle specific directories from the image, and ignore extraneous data.
   • Both implementations could include validation and potentially short-circuit if image contents are not as expected.
5. Is there a way to make a better abstraction for the potential future where we have different source types? Right now, it feels like assumptions about image registries are creeping in.