It seems a breaking change was introduced after v0.6.14

[findmyway]

With v0.6.14

julia> using Flux

julia> NN = NeuralNetworkApproximator(; model = Dense(2, 3), optimizer = Descent())
q_values = NN(rand(2))
NeuralNetworkApproximator{Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, Descent}(Dense(2, 3), Descent(0.1))

julia> q_values = NN(rand(2))
3-element Vector{Float64}:
 -0.15430465457121337
  0.541149992714865
 -0.4218130908450073

julia>         gs = gradient(params(NN)) do
                   sum(NN(rand(2, 5)))
               end
Grads(...)

julia> gs.
grads  params
julia> gs.grads
IdDict{Any, Any} with 2 entries:
  Float32[-0.281172 -0.247735; 0.566611 1.06104; -0.810727 -0.657922] => [2.89303 2.38353; 2.89303 2.38353; 2.89303 2.38353]
  Float32[0.0, 0.0, 0.0]                                              => [5.0, 5.0, 5.0]

With the newest v0.6.16:

julia> using Flux

julia>         NN = NeuralNetworkApproximator(; model = Dense(2, 3), optimizer = Descent())
NeuralNetworkApproximator{Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, Descent}(Dense(2, 3), Descent(0.1))

julia> q_values = NN(rand(2))
3-element Vector{Float64}:
 -0.27442976982279155
  0.046217641920877364
 -0.16964400716726158

julia>         gs = gradient(params(NN)) do
                   sum(NN(rand(2, 5)))
               end
Grads(...)

julia> gs.grads
IdDict{Any, Any} with 2 entries:
  Float32[-0.629986 -0.928219; 0.000942579 0.324789; -0.164906 -0.933509] => [3.37767 2.48145; 3.37767 2.48145; 3.37767 2.48145]
  Float32[0.0, 0.0, 0.0]                                                  => Fill(5.0, 3)

Note the second grads returned a FilledArray instead of a vector.

JuliaReinforcementLearning/ReinforcementLearning.jl#370
cc @pilgrimygy

[CarloLucibello]

I'm not sure which specific PR caused this, but we don't generally commit to the returned gradient type and may provide semantically equivalent alternatives for performance reasons.

Since the Fill optimization in the sum gradient has been there for a while, I'm surprised your code didn't produce fills already in v0.6.14.

Is returning fills problematic for your use case?

[mcabbott]

It does generically consider Fill a valid gradient, e.g. this hasn't changed:

julia> using Zygote

julia> W = [1,2,3]; gradient(() -> sum(W), Params([W]))
Grads(...)

julia> ans[W]
3-element Fill{Int64}, with entries equal to 1

But something is causing this to propagate further than it used to. It might be #1001, but I don't see why right now.

You could also argue that Grads ought always to contain mutable arrays; there is some chance that Flux's optimisers assume that, too.

[findmyway]

You could also argue that Grads ought always to contain mutable arrays; there is some chance that Flux's optimisers assume that, too.

Yeah, that's my concern.

[CarloLucibello]

there is some chance that Flux's optimisers assume that, too.

didn't we fix this?

[findmyway]

there is some chance that Flux's optimisers assume that, too.

didn't we fix this?

I'm afraid not. Most optimizers in https://github.com/FluxML/Flux.jl/blob/master/src/optimise/optimisers.jl will modify Δ in-place.

[ToucheSir]

I believe FluxML/Flux.jl#1613 addresses that, but isn't in a tagged release yet.

[findmyway]

I see. Thanks!

[CarloLucibello]

a new Flux version has been tagged