Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEATURE: Order by emotion on /filter #913

Merged
merged 2 commits into from
Nov 14, 2024
Merged

FEATURE: Order by emotion on /filter #913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
27121d6
FEATURE: Order by emotion on /filter
xfalcox Nov 13, 2024
38d14e4
Address review feedback
xfalcox Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions lib/sentiment/emotion_filter_order.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# frozen_string_literal: true

module DiscourseAi
module Sentiment
class EmotionFilterOrder
def self.register!(plugin)
emotions = %w[
admiration
amusement
anger
annoyance
approval
caring
confusion
curiosity
desire
disappointment
disapproval
disgust
embarrassment
excitement
fear
gratitude
grief
joy
love
nervousness
neutral
optimism
pride
xfalcox marked this conversation as resolved.
Show resolved Hide resolved
realization
relief
remorse
sadness
surprise
]

emotions.each do |emotion|
filter_order_emotion = ->(scope, order_direction) do
emotion_clause = <<~SQL
SUM(
CASE
WHEN (classification_results.classification::jsonb->'#{emotion}')::float > 0.1
THEN 1
ELSE 0
END
)::float / COUNT(posts.id)
SQL
scope
.joins(:posts)
.joins(<<~SQL)
INNER JOIN classification_results
ON classification_results.target_id = posts.id
AND classification_results.target_type = 'Post'
AND classification_results.model_used = 'SamLowe/roberta-base-go_emotions'
SQL
.where(<<~SQL)
topics.archetype = 'regular'
AND topics.deleted_at IS NULL
AND posts.deleted_at IS NULL
AND posts.post_type = 1
SQL
.select(<<~SQL)
topics.*,
#{emotion_clause} AS emotion_#{emotion}
SQL
.group("1")
.having("#{emotion_clause} > 0.05")
.order("#{emotion_clause} #{order_direction}")
end
plugin.add_filter_custom_filter("order:emotion_#{emotion}", &filter_order_emotion)
end
end
end
end
end
2 changes: 2 additions & 0 deletions lib/sentiment/entry_point.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ def inject_into(plugin)
plugin.on(:post_created, &sentiment_analysis_cb)
plugin.on(:post_edited, &sentiment_analysis_cb)

EmotionFilterOrder.register!(plugin)

plugin.add_report("overall_sentiment") do |report|
report.modes = [:stacked_chart]
threshold = 0.6
Expand Down
210 changes: 210 additions & 0 deletions spec/lib/modules/sentiment/emotion_filter_order_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
# frozen_string_literal: true

require "rails_helper"

RSpec.describe DiscourseAi::Sentiment::EmotionFilterOrder do
let(:plugin) { Plugin::Instance.new }
let(:model_used) { "SamLowe/roberta-base-go_emotions" }
let(:post_1) { Fabricate(:post) }
let(:post_2) { Fabricate(:post) }
let(:post_3) { Fabricate(:post) }
let(:classification_1) do
{
love: 0.9444406,
admiration: 0.013724019,
surprise: 0.010188869,
excitement: 0.007888741,
curiosity: 0.006301749,
joy: 0.004060776,
confusion: 0.0028238264,
approval: 0.0018160914,
realization: 0.001174849,
neutral: 0.0008561869,
amusement: 0.00075853954,
disapproval: 0.0006987994,
disappointment: 0.0006166883,
anger: 0.0006000542,
annoyance: 0.0005615011,
desire: 0.00046368592,
fear: 0.00045117878,
sadness: 0.00041727215,
gratitude: 0.00041727215,
optimism: 0.00037112957,
disgust: 0.00035552034,
nervousness: 0.00022954118,
embarrassment: 0.0002049572,
caring: 0.00017737568,
remorse: 0.00011407586,
grief: 0.0001006716,
pride: 0.00009681493,
relief: 0.00008919009,
}
end
let(:classification_2) do
{
love: 0.8444406,
admiration: 0.113724019,
surprise: 0.010188869,
excitement: 0.007888741,
curiosity: 0.006301749,
joy: 0.004060776,
confusion: 0.0028238264,
approval: 0.0018160914,
realization: 0.001174849,
neutral: 0.0008561869,
amusement: 0.00075853954,
disapproval: 0.0006987994,
disappointment: 0.0006166883,
anger: 0.0006000542,
annoyance: 0.0005615011,
desire: 0.00046368592,
fear: 0.00045117878,
sadness: 0.00041727215,
gratitude: 0.00041727215,
optimism: 0.00037112957,
disgust: 0.00035552034,
nervousness: 0.00022954118,
embarrassment: 0.0002049572,
caring: 0.00017737568,
remorse: 0.00011407586,
grief: 0.0001006716,
pride: 0.00009681493,
relief: 0.00008919009,
}
end
let(:classification_3) do
{
anger: 0.8503682,
annoyance: 0.08113059,
disgust: 0.020593312,
disapproval: 0.013718102,
neutral: 0.0074148285,
disappointment: 0.005785964,
sadness: 0.0028253668,
curiosity: 0.0028253668,
confusion: 0.0023885092,
surprise: 0.001524171,
embarrassment: 0.0012784768,
love: 0.001177788,
admiration: 0.0010892758,
realization: 0.001080799,
approval: 0.00102328,
fear: 0.00097261387,
amusement: 0.0007724123,
excitement: 0.00059921003,
gratitude: 0.00055852515,
joy: 0.00054986606,
optimism: 0.00050458545,
desire: 0.00046849172,
caring: 0.00037205798,
remorse: 0.00028415458,
grief: 0.00025973833,
nervousness: 0.00024305031,
pride: 0.00011661681,
relief: 0.00007470753,
}
end
let!(:classification_result_1) do
Fabricate(
:sentiment_classification,
target: post_1,
model_used: model_used,
classification: classification_1,
)
end
let!(:classification_result_2) do
Fabricate(
:sentiment_classification,
target: post_2,
model_used: model_used,
classification: classification_2,
)
end
let!(:classification_result_3) do
Fabricate(
:sentiment_classification,
target: post_3,
model_used: model_used,
classification: classification_3,
)
end

before { described_class.register!(plugin) }

it "registers emotion filters" do
emotions = %w[
disappointment
sadness
annoyance
neutral
disapproval
realization
nervousness
approval
joy
anger
embarrassment
caring
remorse
disgust
grief
confusion
relief
desire
admiration
optimism
fear
love
excitement
curiosity
amusement
surprise
gratitude
pride
]

filters = DiscoursePluginRegistry.custom_filter_mappings.reduce(Hash.new, :merge)

emotions.each { |emotion| expect(filters).to include("order:emotion_#{emotion}") }
end

it "filters topics by emotion" do
Copy link
Contributor

@pmusaraj pmusaraj Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I am confused a bit about ordering vs filtering here. The admin report for a given emotion should point to a filtered list of topics where there is a match for that emotion, right? For that, would we need an explicit filter like emotion:disgust?

We certainly want order:emotion_disgust, no doubt, but I think we also need the filtering...

Copy link
Member Author

@xfalcox xfalcox Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will all be accessed from a link, they aren't supposed to be typed, so do we really need the double emotion:disgust order:emotion_disgust ?

Filter and Order here are a bit fuzzy. Ordering works, you simply show first topics where most posts have that emotion. But filter means you need to set a threshold, as every classification contains some of that emotion as an infinitesimal decimal. What do you suggest? Also worth noting that emotion is per post, while filter acts on topics, so we need a way to translate the concepts.

I'd go with a having clause limiting topics to where the emotion is presented with a score of at least 10% in at least 5% of the replies built-in into the ordering, at least for starters. Expect those to be adjusted over time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, not a fan of the double keywords. Not a fan of order:X filtering out results either because it goes against what order does in other places. I don't have a definitive answer, maybe in the short-run we go with the order: keyword only and accept it is exceptional.

Regarding the cutoff, I'm happy with your suggestion. The main thing is that we need to use the same threshold that we use in the admin report:

image

The graph here will likely change to accomodate the additional emotion dimensions, but, if it will still show a count per emotion, we need to respect that same count in the filtered view.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That graph is going the way of the dodo. We will replace it with a table that will contain the links to this new filter.

This PR is only adding the filter, but they won't be used yet.

Added the proposed filter in 38d14e4, see the updated tests.

emotion = "joy"
scope = Topic.all
order_direction = "desc"

filter =
DiscoursePluginRegistry
.custom_filter_mappings
.find { _1.keys.include? "order:emotion_#{emotion}" }
.values
.first
result = filter.call(scope, order_direction)

expect(result.to_sql).to include("INNER JOIN classification_results")
expect(result.to_sql).to include(
"classification_results.model_used = 'SamLowe/roberta-base-go_emotions'",
)
expect(result.to_sql).to include("topics.archetype = 'regular'")
expect(result.to_sql).to include("ORDER BY")
expect(result.to_sql).to include("->'#{emotion}'")
expect(result.to_sql).to include("desc")
end

it "sorts emotion in ascending order" do
expect(
TopicsFilter
.new(guardian: Guardian.new)
.filter_from_query_string("order:emotion_love-asc")
.pluck(:id),
).to contain_exactly(post_2.topic.id, post_1.topic.id)
end
it "sorts emotion in default descending order" do
expect(
TopicsFilter
.new(guardian: Guardian.new)
.filter_from_query_string("order:emotion_love")
.pluck(:id),
).to contain_exactly(post_1.topic.id, post_2.topic.id)
end
end