Introduce "collection" projects for better usage of hierarchical view #2041 #3258

rkg-mm · 2023-12-02T16:18:16Z

Description

This change introduces logic for "collection projects". Those are basically projects used as parent for other projects that shall not hold any own component or vulnerability data, but instead get calculated from child projects using different configurable aggregation logics.

Corresponding Frontend PR with Screenshots: DependencyTrack/frontend#658

Addressed Issue

resolves #2041
resolves #657
resolves #2410

Additional Details

Checklist

I have read and understand the contributing guidelines
~~This PR fixes a defect, and I have provided tests to verify that the fix is effective~~
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have added corresponding update logic
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

cheapshot2000 · 2024-03-25T21:52:24Z

Has there been any updates/progress on the feature in completion of the 4.11 release?

TWpgo · 2024-04-22T11:34:35Z

Looking forward for this feature 👍

rkg-mm · 2024-04-23T08:56:17Z

@nscuro I wonder why the unit test still fails. I don't see a change that could influence the parent of a project not being serialized?

nscuro · 2024-04-23T09:10:50Z

@rkg-mm FWIW there can be subtle changes in how serialization libraries (i.e. Jackson) handle certain fields, when those libraries are updated. Or how the persistence framework fetches fields implicitly.

rkg-mm · 2024-04-23T09:33:16Z

Yea so, the PRs backend & frontend seem to be working well. But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

 Error:  Tests run: 40, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 74.367 s <<< FAILURE! - in org.dependencytrack.resources.v1.ProjectResourceTest
Error:  patchProjectParentTest(org.dependencytrack.resources.v1.ProjectResourceTest)  Time elapsed: 1.483 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: 
JSON documents are different:
Different keys found in node "", missing: "parent", expected: 
<{"active":true,"collectionLogic":"NONE","name":"DEF","parent":{"name":"GHI","uuid":"${json-unit.matches:parentProjectUuid}","version":"3.0"},"properties":[],"tags":[],"uuid":"${json-unit.matches:projectUuid}","version":"2.0"}> 
but was: 
<{"active":true,"collectionLogic":"NONE","name":"DEF","properties":[],"tags":[],"uuid":"ba033fa5-aa20-44ae-85de-6a33e5cf36a3","version":"2.0"}>

	at org.dependencytrack.resources.v1.ProjectResourceTest.patchProjectParentTest(ProjectResourceTest.java:733)

nscuro · 2024-04-28T13:01:51Z

src/main/java/org/dependencytrack/model/ProjectCollectionLogic.java

+ * @author Ralf King
+ * @since 4.11.0
+ */
+public enum ProjectCollectionLogic {


It would be good to add some documentation about this. In #2041 you mention a few use cases that these address, having those be part of the docs will definitely help others.

Docs added. However, since I'm stuck to Windows and there is no build script for the docs in windows, I could not really test if this integrates properly. Can you check this?

src/main/java/org/dependencytrack/persistence/ProjectQueryManager.java

nscuro · 2024-04-28T13:24:06Z

src/main/java/org/dependencytrack/persistence/ProjectQueryManager.java

@@ -538,6 +547,7 @@ public Project updateProject(Project transientProject, boolean commitIndex) {
        project.setDescription(transientProject.getDescription());
        project.setVersion(transientProject.getVersion());
        project.setClassifier(transientProject.getClassifier());
+        project.setCollectionLogic(transientProject.getCollectionLogic());


Changing the collection logic from say AGGREGATE_DIRECT_CHILDREN to HIGHEST_SEMVER_CHILD can drastically change metric values on existing projects. There will be no indication as to if and when the logic was changed, which can make it tricky to explain why values changed so much.

That's correct, but also it would be an intended change. Idk what could be done about this except add such change to the logfile, but that would only be helpful for deep analysis of such issue.

Fair.

Naive idea: We could add a Collection logic change flag to ProjectMetrics. We could use that in the metrics graph, in some similar way to what you can do in Grafana with annotations:

I like that idea, but at least on a first look the frontend chart lib doesn't support this. not sure how difficult this would be to add on top of it. Any experience with that? I never used that component.
From data perspective in backend it should not be too complicated.

edit: or would be adding it to the charts tooltip be enough? Also it seems there are options to style datapoints differently. Maybe we could give such points a different styling + add the info in the tooltip?

I don't have too strong opinions on how we highlight it, but I do think that highlighting it one way or the other would be good.

Tooltip on it's own might be too hidden, but styling the datapoints sounds good.

@nscuro would you be fine with creating a new issue to add this visibility later as further enhancement, and for now rely on log file for visibility in case someone wonders? we could also add a hint in the project edit UI when someone changes the logic, that informs him of this for now?

Logging is probably fine for an MVP implementation, yes. But ultimately we will really need some kind of indicator in the UI.

@nscuro I found a way to show it in the metrics chart. Only issue is that the line gets invisible when hovered. No Idea why. But I think this is fine as it is, looks like its intended :D

Ok found the reason for the visibility issue and fixed it. Also extended it to the other charts now. Looks like this:

Hint: The blue line at 0 will no longer be there, I improved the backend to not record the initial entry after an update to the new version anymore as changed collection logic, my DB just has these entries already

src/main/java/org/dependencytrack/tasks/metrics/ProjectMetricsUpdateTask.java

src/main/java/org/dependencytrack/model/ProjectCollectionLogic.java

nscuro · 2024-04-28T14:07:38Z

@rkg-mm But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

I debugged the area in question multiple times now and still don't quite understand how this is happening. The parent is part of the response when I comment out these lines: https://github.com/DependencyTrack/dependency-track/pull/3258/files#diff-75cd29a0a085d84dff3cb4794242e0d1b400385f1acfae172346c3e6780629b0R582-R590

Something about those lines causes parent to be unloaded. But they're literally just calling a setter, so I don't quite get why. The test succeeds when adding project.getParent(); after the lines mentioned above. This kind of "force loading" is used in other areas of the code base. I don't like it, but in this case I think it's OK to do it.

rkg-mm · 2024-05-05T20:46:36Z

oh come on.. Idk what IntelliJ is doing but whenever I rebase this branch on your master I end up having all changes from master in my PR again. What am I doing wrong here? I don't get it...

edit: ok fixed it again with some hacky workaround. still don't know what I do wrong here...

nscuro · 2024-05-05T20:54:08Z

@rkg-mm How do you do the rebase? I usually run git pull --rebase upstream master in the command line and then use IntelliJ to resolve any conflicts in case there are any.

I never had this particular problem happen, but can imagine IntelliJ might do some unexpected things when merging/rebasing via UI.

rkg-mm · 2024-05-05T20:58:58Z

In IntelliJ UI I select the upstream master and select "Rebase (my branch) onto master" then push the changes to my remote. The strange thing happens here already, as suddenly my own remote has changes not recognized by my local branch and needs to either be merged (which I tried before and ended up with double changes on same files) or I select "rebase anyway" which avoids double merging. But in both cases I still end up with all those unwanted changes again. Maybe I should try the console attempt next time...

rkg-mm · 2024-05-06T16:16:16Z

@rkg-mm But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

I debugged the area in question multiple times now and still don't quite understand how this is happening. The parent is part of the response when I comment out these lines: https://github.com/DependencyTrack/dependency-track/pull/3258/files#diff-75cd29a0a085d84dff3cb4794242e0d1b400385f1acfae172346c3e6780629b0R582-R590

Something about those lines causes parent to be unloaded. But they're literally just calling a setter, so I don't quite get why. The test succeeds when adding project.getParent(); after the lines mentioned above. This kind of "force loading" is used in other areas of the code base. I don't like it, but in this case I think it's OK to do it.

Fixed by your suggestion

codacy-production · 2024-05-06T16:36:32Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.10% (target: -1.00%)	✅ 86.04% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`ff22d4f`)	22069	16756	75.93%
Head commit (`0cfdd8c`)	22259 (+190)	16923 (+167)	76.03% (+0.10%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3258)	222	191	86.04%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy will stop sending the deprecated coverage status from June 5th, 2024. Learn more}

nscuro · 2024-05-07T14:12:39Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

rkg-mm · 2024-05-07T16:04:35Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

no problem, I understand. Lets just hope the release cycle is a bit shorter this time :D

cheapshot2000 · 2024-05-07T16:06:22Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

no problem, I understand. Lets just hope the release cycle is a bit shorter this time :D

Is there a set release cadence? or other logic for determining when releases are completed?

nscuro · 2024-05-07T16:18:09Z

Is there a set release cadence? or other logic for determining when releases are completed?

There is not at the moment. We originally planned to do monthly releases, but ended up stretching that (by far...) yet again. A problem is that the core team has certain items they want (and have to work on) in a given release, while said core team also needs to review and test community contributions. It's a luxury problem, but it means that more time is needed to meet both our own, and the community's expectations.

This particular PR was planned for inclusion in 4.11 for a long time, but it also touches on the very core of how DT behaves, in turn requiring more review and testing than other, smaller PRs. And of course contributors also don't always have time to answer our annoying requests. :)

We're still in the process of figuring out how to best plan and size releases. Going forward, we're labelling issues with T-shirt sizes, according to their effort. Aim being to limit the number of high-effort items per release, enabling faster cycles.

msymons · 2024-08-08T08:26:47Z

Re-assigning to 4.13 milestone in order to reduce the pressure on contributors to get this finished whilst allowing v4.12.0 to be released quicker.

codacy-production · 2024-09-14T16:37:29Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.09% (target: -1.00%)	✅ 88.74% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`6f7c49c`)	22460	17770	79.12%
Head commit (`2bf56f0`)	22649 (+189)	17941 (+171)	79.21% (+0.09%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3258)	222	197	88.74%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

rkg-mm · 2024-09-16T16:31:54Z

@nscuro apart from the one proposal in the UI PR which I added some question for, this should be finished now. Can you review it again please?

netomi · 2024-10-02T18:19:08Z

Nice to see this PR moving forward. We run a previous version of the PR in our experimental setup. Will update with the latest version and provide feedback.

Some minor thing that we noticed and that would be great if that could be changed or supported as well: when modeling your products in a parent / child relationship like that:

You are forced to set a classifier for the parent project, though it is purely a project container and the classifier does not make any sense in this context. If the classifier could be made optional or better reflect the concept of a project container, it would be nice and avoid confusion when looking at the project hierarchy.

rkg-mm · 2024-10-03T09:55:48Z

Nice to see this PR moving forward. We run a previous version of the PR in our experimental setup. Will update with the latest version and provide feedback.

Some minor thing that we noticed and that would be great if that could be changed or supported as well: when modeling your products in a parent / child relationship like that:

You are forced to set a classifier for the parent project, though it is purely a project container and the classifier does not make any sense in this context. If the classifier could be made optional or better reflect the concept of a project container, it would be nice and avoid confusion when looking at the project hierarchy.

So basically you want to have an option which marks the parent as a collection project, hence removes the component uploads etc. but does not calculate any values based on children? what should be displayed inside the parent then? nothing?

…CollectionLogics. Modifies project metrics updates to consider collection project logics. Added several events to trigger project metrics updates to update collection projects if child projects change in a relevant way. Signed-off-by: Ralf King <[email protected]>

Add test cases for new metrics update logic for collection projects. Signed-off-by: Ralf King <[email protected]>

… all projects Signed-off-by: Ralf King <[email protected]>

Signed-off-by: Ralf King <[email protected]>

… wrongly influence numbers Signed-off-by: Ralf King <[email protected]>

Signed-off-by: Ralf King <[email protected]>

* Add validation to prevent invalid states of collection projects (prevent collection project having Components or services), including several unit tests covering these scenarios. Signed-off-by: Ralf King <[email protected]>

…ject actually changed. Signed-off-by: Ralf King <[email protected]>

Signed-off-by: Ralf King <[email protected]>

…" flag in projects (to be implemented with issue 4148) Signed-off-by: Ralf King <[email protected]>

…his feature Signed-off-by: Ralf King <[email protected]>

…nt in UI to visualize why drastic changes in numbers occured. Signed-off-by: Ralf King <[email protected]>

…change in UI for first new entry. Signed-off-by: Ralf King <[email protected]>

Signed-off-by: Ralf King <[email protected]>

netomi · 2024-10-03T11:32:46Z

so the behavior is just fine as it is right now. It feels just odd that when modeling your projects with dependency track you are forced to enter some data for a project whose only purpose is to act as a container for nested projects.

Right now you have to select at least a classifier afaict, so in our case (see the screenshot) we have selected Application, though it is not an Application, but the project that consists of various software products. When selecting that project, you should see all vulnerabilities of child projects as done right now.

This was discussed yesterday in the community call and its a really small thing but people asked me to raise it in this ticket as well.

Signed-off-by: Ralf King <[email protected]>

…_VERSION_CHILDREN to match other 2 names * Add missing test for AGGREGATE_LATEST_VERSION_CHILDREN logic * Update outdated tests with additional property Signed-off-by: Ralf King <[email protected]>

rkg-mm · 2024-10-04T13:10:12Z

@nscuro
From my point of view this PR is good to go now, would be great if you could review this before I have a lot of conflicts to fix again ;-)
Are you fine with making the classifier optional (always or only in case of collections?) as requested by @netomi ? I missed the community meeting and the recording is not there yet, so I cannot check what was said there.

Signed-off-by: Ralf King <[email protected]>

rkg-mm · 2024-10-06T00:28:24Z

@nscuro From my point of view this PR is good to go now, would be great if you could review this before I have a lot of conflicts to fix again ;-) Are you fine with making the classifier optional (always or only in case of collections?) as requested by @netomi ? I missed the community meeting and the recording is not there yet, so I cannot check what was said there.

I added a NONE classifier now after watching the community meetings on youtube.
I furthermore noticed that Frontend does not support all classifiers supported by backend so far, so I also added support for the missing classifiers in Frontend.

codacy-production · 2024-10-06T00:36:18Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.09% (target: -1.00%)	✅ 88.79% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`6f7c49c`)	22460	17770	79.12%
Head commit (`f14fc9a`)	22710 (+250)	17988 (+218)	79.21% (+0.09%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3258)	223	198	88.79%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

netomi · 2024-10-15T15:33:59Z

fyi: a package of this PR is available at https://github.com/users/netomi/packages/container/dtrack-apiserver/289607235?tag=4.13.0-SNAPSHOT

ghcr.io/netomi/dtrack-apiserver:4.13.0-SNAPSHOT

rkg-mm mentioned this pull request Dec 2, 2023

Introduce "collection" projects for better usage of hierarchical view #2041 DependencyTrack/frontend#658

Open

2 tasks

nscuro added this to the 4.11 milestone Dec 3, 2023

nscuro self-assigned this Jan 11, 2024

nscuro mentioned this pull request Jan 15, 2024

Add optional organization/group structure #657

Open

rkg-mm force-pushed the 2041-introduce-collection-projects branch from 0aa204e to ddc5c2b Compare April 23, 2024 00:50

nscuro reviewed Apr 28, 2024

View reviewed changes

rkg-mm force-pushed the 2041-introduce-collection-projects branch from efe2c29 to ecfc394 Compare May 5, 2024 20:53

nscuro modified the milestones: 4.11, 4.12 May 7, 2024

msymons modified the milestones: 4.12, 4.13 Aug 8, 2024

rkg-mm force-pushed the 2041-introduce-collection-projects branch from 0cfdd8c to cb0b75b Compare September 14, 2024 15:30

rkg-mm mentioned this pull request Sep 14, 2024

Introduce isLatest flag to mark latest version of project #4148

Closed

2 tasks

rkg-mm added 14 commits October 3, 2024 12:01

Fix missing default.

5b559ea

Add test cases for new metrics update logic for collection projects. Signed-off-by: Ralf King <[email protected]>

Add upgrade class to set default project collection logic to NONE for…

61849be

… all projects Signed-off-by: Ralf King <[email protected]>

Fix unit test

1a5af6f

Signed-off-by: Ralf King <[email protected]>

Exclude collection projects from portfolio metrics calculation to not…

b5fd40b

… wrongly influence numbers Signed-off-by: Ralf King <[email protected]>

Updated copyright

e4f2a64

Signed-off-by: Ralf King <[email protected]>

Add new Property to unit test data

f3b74c3

Signed-off-by: Ralf King <[email protected]>

* Fix failing unit test by enforcing parent loading

5f5c7ab

* Add validation to prevent invalid states of collection projects (prevent collection project having Components or services), including several unit tests covering these scenarios. Signed-off-by: Ralf King <[email protected]>

Only dispatch parent metrics update event when metrics of current pro…

84698c0

…ject actually changed. Signed-off-by: Ralf King <[email protected]>

Fix unit tests

aed9e55

Signed-off-by: Ralf King <[email protected]>

Remove HighestSemverChild logic, replace by preparation for "isLatest…

dfada3e

…" flag in projects (to be implemented with issue 4148) Signed-off-by: Ralf King <[email protected]>

Move upgrade logic to v4.13.0 as this is the new target version for t…

8bd3e1a

…his feature Signed-off-by: Ralf King <[email protected]>

Add collection logic changes to project metrics for further improveme…

1d81af0

…nt in UI to visualize why drastic changes in numbers occured. Signed-off-by: Ralf King <[email protected]>

Improve handling of old Metrics DB entries to avoid showing a metric …

5fc4375

…change in UI for first new entry. Signed-off-by: Ralf King <[email protected]>

rkg-mm force-pushed the 2041-introduce-collection-projects branch from 5acc770 to 5fc4375 Compare October 3, 2024 10:09

rkg-mm added 2 commits October 3, 2024 12:17

Fix merge issue

5ad555c

Signed-off-by: Ralf King <[email protected]>

Enable logic for collection projects using isLatest flag

8bf4556

Signed-off-by: Ralf King <[email protected]>

rkg-mm added 2 commits October 4, 2024 14:06

Add usage example in docs for collection projects.

cd07dcc

Signed-off-by: Ralf King <[email protected]>

* Rename LATEST_VERSION_CHILDREN collection logic to AGGREGATE_LATEST…

2bf56f0

…_VERSION_CHILDREN to match other 2 names * Add missing test for AGGREGATE_LATEST_VERSION_CHILDREN logic * Update outdated tests with additional property Signed-off-by: Ralf King <[email protected]>

Allow setting NONE as classifier (useful for parents)

f14fc9a

Signed-off-by: Ralf King <[email protected]>

Introduce "collection" projects for better usage of hierarchical view #2041 #3258

Are you sure you want to change the base?

Introduce "collection" projects for better usage of hierarchical view #2041 #3258

Conversation

rkg-mm commented Dec 2, 2023 • edited Loading

Description

Addressed Issue

Additional Details

Checklist

cheapshot2000 commented Mar 25, 2024

TWpgo commented Apr 22, 2024

rkg-mm commented Apr 23, 2024

nscuro commented Apr 23, 2024

rkg-mm commented Apr 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkg-mm May 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkg-mm Aug 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkg-mm Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

nscuro commented Apr 28, 2024

rkg-mm commented May 5, 2024 • edited Loading

nscuro commented May 5, 2024

rkg-mm commented May 5, 2024

rkg-mm commented May 6, 2024

codacy-production bot commented May 6, 2024

Coverage summary from Codacy

nscuro commented May 7, 2024

rkg-mm commented May 7, 2024

cheapshot2000 commented May 7, 2024

nscuro commented May 7, 2024

msymons commented Aug 8, 2024

codacy-production bot commented Sep 14, 2024 • edited Loading

Coverage summary from Codacy

rkg-mm commented Sep 16, 2024

netomi commented Oct 2, 2024 • edited Loading

rkg-mm commented Oct 3, 2024

netomi commented Oct 3, 2024

rkg-mm commented Oct 4, 2024

rkg-mm commented Oct 6, 2024

codacy-production bot commented Oct 6, 2024

Coverage summary from Codacy

netomi commented Oct 15, 2024 • edited Loading

rkg-mm commented Dec 2, 2023 •

edited

Loading

rkg-mm May 24, 2024 •

edited

Loading

rkg-mm Aug 11, 2024 •

edited

Loading

rkg-mm Sep 16, 2024 •

edited

Loading

rkg-mm commented May 5, 2024 •

edited

Loading

codacy-production bot commented Sep 14, 2024 •

edited

Loading

netomi commented Oct 2, 2024 •

edited

Loading

netomi commented Oct 15, 2024 •

edited

Loading