English words in a sentence wrongly identify the language as english

[sxilderik]

« Ce connard impose un meeting sur le rooftop asap » which is correctly identified as French by translate.google.com is identified as English by lingua.

I plan to use speech recognition to apply GDPR on french texts only. Lingua would wrongly ignore that sentence, which is a no go for me…

NB: this sentence is definitely french, those english words are – alas – of common use in business or office French.

[pemistahl]

Hi @sxilderik, thank you for opening this discussion. Well, the statistical approach of my library which uses relative frequencies of character ngrams is sometimes misguided. Mixed-language texts are the hardest to identify correctly. I'm sure that Google additionally uses dictionaries to identify certain words which explains why Google succeeds and Lingua fails here.

But it turns out that the summed and logarithmized probabilities for the different ngram lengths are pretty close to each other:

Ngram Length	English	French
1	-48.99	-48.91
2	-83.97	-94.79
3	-64.48	-71.86
4	-33.27	-40.93
5	-16.19	-16.00

For ngrams of length 1 and 5, French is more likely than English. But for ngrams of length 2, 3 and 4, English is more likely than French. The French words in your sentence do not exhibit any specific ngrams that are typical of French. There are no accents either. If those were there, I'm sure the detector would decide in favor of French.

What is your overall experience with my library? Do you have a lot of incorrectly detected sentences?

[sxilderik]

Thanks for your reply!
Actually, I would be satisfied by those numbers if I could get them.
My plan is to avoid analyzing non-french texts for unacceptable (french) phrases. I dont need 100% accuracy in language detection. If I can identity a block of text of being french with a reasonable confidence, I would be happy.
Right now, as far as I know, the only result I get is a language name, without any clue as how confident I can be in this result.

But maybe I am mistaken, and this info is already available somewhere I did not look.
If not, I think you should consider delivering this info.

[sxilderik]

Well, I found the API computeLanguageConfidenceValues. I use this one now, keeping only the languages with 90% confidence or more. If one of these languages is French, then I consider the sentence for further testings.
Thanks !