View Issue Details

IDProjectCategoryView StatusLast Update
0011389ScribusLanguage Toolspublic2014-07-07 15:42
ReporterJLuc Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
OSubuntuOS Version12.04 
Product Version1.4.2 
Summary0011389: spellchecker find wrong mistakes
DescriptionWith new 1.4.2 on ubuntu 12.04, with libhunspell 1.3.2-4 and hunspell-fr 1:3.3.0-2ubuntu3,

The spell checker find wrong errors, all related to "qu'" words : lorsqu', quoiqu', qu'




Steps To Reproduce- create new doc
- F3 and change default character style langage to french
- T so as to create text frame,
- insert default french false text (french Lorem Ipsum) in text frame,
- F2 and set this text style to default paragraphe style and default character style
- select text frame
- SHIFT F7 so as to launch spellchecker

see the wrong errors.
Additional InformationSame spellchecker in libreoffice with same pasted text does not raise these non-mistakes. Both scribus and libreoffice use same dictionnary (checked in uninstalling it).

Maybe related to some local issue ?
TagsNo tags attached.
Patch

Activities

JLuc

2013-01-21 13:57

developer  

test spellchecker.sla (18,783 bytes)

JLuc

2013-01-21 13:57

developer   ~0029748

I uploaded test file.

cbradney

2013-02-05 20:37

administrator   ~0029793

Pasting this in for reference: http://blogs.msdn.com/b/correcteurorthographiqueoffice/archive/2005/12/07/identifying-tokens-is-word-breaking-so-easy.aspx

Right now we break words on !isLetter(), but changing that will have all kinds of deep rooted effects, so this is not just a spelling issue.

cbradney

2013-02-05 20:38

administrator   ~0029794

Basically:

int StoryText::endOfWord(int pos) const
{
    int len = length();
    while (pos < len)
    {
        if(text(pos).isLetter())
            ++pos;
        else
            break;
    }
    return pos;
}

It makes no consideration of language or script

JLuc

2013-02-12 19:28

developer   ~0029833

Amof, there are no problems with other uses of ' as in "c'est" or "L'oiseau" : no error is detected and that's fine. It's only with "anythinQU'xxx" that an error is mistakenly rised.

So i'm afraid that changing the ' status for french will cure the "qu'" issue but might result in problems for other situations like "c'est" where as is code does fine now.

nemeth

2013-02-12 20:37

reporter   ~0029834

It seems, it is a (1) tokenization and/or (2) Unicode (older Hunspell dictionary) problem.

(1) Hunspell with the French dictionary recognizes the qu' words only in their full forms, eg. in the command line:

$ echo "lorsqu'un" | hunspell -d fr
+ un

The newest French dictionary recognizes the ASCII and the true apostrophe (U+2019 = ’), too:

$ echo "lorsqu’un" | hunspell -d fr
+ un

Older hunspell-fr dictionaries don't specify the apostrophe as a WORDCHARS in its affix file, but you can check with the following input with tabulator:

$ echo "lorsqu'un " | hunspell -d fr -1
+ un)

(2) Older Hunspell dictionaries doesn't recognize the words with true (U+2019) apostrophes. It is not problem with LibreOffice, because it converts them to ASCII apostrophes before calling Hunspell. The recent French Hunspell dictionary of LibreOffice has already contained the following line in the ICONV section of the aff file:

ICONV ’ '
 
So if Scribus calls Hunspell with the right tokenization, but U+2019 apostrophes, system-wide French Hunspell dictionary may need an update. Other option to convert U+2019 characters to ASCII characters in Scribus.

By the way, spell checking component of LibreOffice just got a similar fix to handle poor man's Unicode f-ligatures and zero width non-joiner Unicode characters (ZWNJ could be important to fix German typography with automatic f-ligature replacements):

http://cgit.freedesktop.org/libreoffice/core/commit/?id=98029f1625663609d670f79eea61f7547bfc8123

olivierr

2013-02-13 00:27

reporter   ~0029835

> Right now we break words on !isLetter(), but changing that will have all kinds of deep rooted effects, so this is not just a spelling issue.

If the apostrophe has to be considered as a word break, you have to patch the dictionary to add the following words (elided forms):

- lorsqu
- puisqu
- jusqu
- puisqu
- quoiqu
- aujourd

That would solve the main issues.

(Maybe I will add these elided forms in the upstream dictionary.)

Then you could also add splitted words from every word that contains an apostrophe.

olivierr

2013-02-17 09:45

reporter   ~0029850

Last edited: 2013-02-17 09:50

I added in the Hunspell dictionary the following elided forms:
— lorsqu : http://www.dicollecte.org/entry.php?prj=fr&id=222035
— puisqu : http://www.dicollecte.org/entry.php?prj=fr&id=222036
— jusqu : http://www.dicollecte.org/entry.php?prj=fr&id=222037
— quoiqu : http://www.dicollecte.org/entry.php?prj=fr&id=222038
— quelqu : http://www.dicollecte.org/entry.php?prj=fr&id=222039

That should solve most of the issues with apostrophes.

I did it only because they are valid elided forms, but I won’t add other word fragments in the dictionary.

The new Hunspell dictionary will be released in march or april 2013.

Kunda

2014-07-06 23:17

updater   ~0032518

jluc, not able to reproduce on 1.5svn r19322

JLuc

2014-07-07 09:27

developer   ~0032531

Hmmm i still reproduce.
Try with following sentence :

"le bug se produit mais l'affichage est foireux lorsqu'on met le mot au tout début du frame"

JLuc

2014-07-07 12:01

developer   ~0032533

my hunspell-fr version is 1.3.3.0-2ubuntu3

i have to try with newer version

wait and see - status for this report is now 'commentaire'

JLuc

2014-07-07 12:12

developer   ~0032535

Fixed with new hunspell-fr version found http://www.dicollecte.org/download.php?prj=fr

FirasH

2014-07-07 15:42

reporter   ~0032547

As fixed we can close it :)

Issue History

Date Modified Username Field Change
2013-01-21 13:41 JLuc New Issue
2013-01-21 13:57 JLuc File Added: test spellchecker.sla
2013-01-21 13:57 JLuc Note Added: 0029748
2013-02-05 20:37 cbradney Note Added: 0029793
2013-02-05 20:38 cbradney Note Added: 0029794
2013-02-12 19:28 JLuc Note Added: 0029833
2013-02-12 20:37 nemeth Note Added: 0029834
2013-02-13 00:27 olivierr Note Added: 0029835
2013-02-17 09:45 olivierr Note Added: 0029850
2013-02-17 09:50 olivierr Note Edited: 0029850
2014-07-06 23:17 Kunda Note Added: 0032518
2014-07-06 23:17 Kunda Status new => feedback
2014-07-07 09:27 JLuc Note Added: 0032531
2014-07-07 09:27 JLuc Status feedback => new
2014-07-07 12:01 JLuc Note Added: 0032533
2014-07-07 12:12 JLuc Note Added: 0032535
2014-07-07 12:12 JLuc Status new => resolved
2014-07-07 12:12 JLuc Resolution open => fixed
2014-07-07 12:12 JLuc Assigned To => JLuc
2014-07-07 15:42 FirasH Note Added: 0032547
2014-07-07 15:42 FirasH Status resolved => closed
2014-07-07 15:42 FirasH Assigned To JLuc =>