View Issue Details

IDProjectCategoryView StatusLast Update
0017367ScribusTypographypublic2025-01-09 21:28
Reporterjirib Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version1.6.3.svn 
Summary0017367: [hyphenator.cpp] hyphenation works differently even when using same hyphenation dict
DescriptionThis is how is the Czech word 'orgánu' hyphenated in Scribus:

$
$ grep -Po ' \K(or.*nu)\s+' pl-reprint-01.sla | xxd
00000000: 6f72 c2ad 67c3 a16e 7520 0a or..g..nu .

# in literal: or-gánu ('-' is soft hyphen)

A proof that same hyphen dict is used:

$ md5sum ~/.local/share/scribus/dicts/hyph/hyph_cs_CZ.dic /home/jiri/.venv3/lib/python3.11/site-packages/pyphen/dictionaries/hyph_cs_CZ.dic
c9cbd906b7884d88b00a3a0a6a7773c7 /home/jiri/.local/share/scribus/dicts/hyph/hyph_cs_CZ.dic
c9cbd906b7884d88b00a3a0a6a7773c7 /home/jiri/.venv3/lib/python3.11/site-packages/pyphen/dictionaries/hyph_cs_CZ.dic

With pyphen:

>>> import pyphen
>>> dic = pyphen.Pyphen(lang='cs_CZ')
>>> text = 'orgánu'
>>> ' '.join(dic.inserted(word) for word in text.split())
'or-gá-nu'

There's a TODO note in the code regarding 'hnj_hyphen_hyphenate2()' - https://github.com/scribusproject/scribus/blob/master/scribus/hyphenator.cpp#L116 .

Is that the reason?

The "impact" is unpleasant rivers of white :-(
Steps To ReproduceSee example above.
TagsNo tags attached.
PatchNo

Activities

jirib

2025-01-08 18:57

reporter   ~0051889

Eh, pyphen is python-only implementation, it does not use hunspell library. So, I don't know if this is a bug or not :/

jirib

2025-01-08 19:15

reporter   ~0051890

s/hunspell/hyphen/ - i'm very sorry, i'm new to this.

a test via C code from hyphen:

$ gcc -o hyphen_test /tmp/hyphen.c -lhyphen
$ ./hyphen_test
Hyphenation points for 'orgánu':
Hyphenation point at index 1
Hyphenation point at index 4
Hyphenated word: or-gá-nu

jirib

2025-01-08 19:15

reporter   ~0051891

C code for hyphen_test
hyphen.c (1,361 bytes)   
#include <stdio.h>
#include <string.h>
#include "hyphen.h"

int main() {
    HyphenDict *hd = hnj_hyphen_load("hyph_cs_CZ.dic");
    if (!hd) {
        fprintf(stderr, "Failed to load hyphenation patterns\n");
        return 1;
    }

    const char *word = "orgánu";
    int word_size = strlen(word);

    char hyphens[word_size + 5];
    memset(hyphens, 0, sizeof(hyphens));

    char hyphenated_word[word_size * 2]; 
    memset(hyphenated_word, 0, sizeof(hyphenated_word));

    char **rep = NULL; 
    int *pos = NULL;   
    int *cut = NULL;   

    int result = hnj_hyphen_hyphenate2(hd, word, word_size, hyphens,
                                       hyphenated_word, &rep, &pos, &cut);

    if (result != 0) {
        fprintf(stderr, "Hyphenation failed\n");
        hnj_hyphen_free(hd);
        return 1;
    }

    int index = 0;
    for (int i = 0; i < word_size; i++) {
        hyphenated_word[index++] = word[i];
        if (hyphens[i] % 2 != 0) { 
            hyphenated_word[index++] = '-';
        }
    }
    hyphenated_word[index] = '\0'; 

    printf("Hyphenation points for '%s':\n", word);
    for (int i = 0; i < word_size; i++) {
        if (hyphens[i] % 2 != 0) {
            printf("Hyphenation point at index %d\n", i);
        }
    }
    printf("Hyphenated word: %s\n", hyphenated_word);

    hnj_hyphen_free(hd); 
    return 0;
}
hyphen.c (1,361 bytes)   

cbradney

2025-01-08 21:59

administrator   ~0051892

Scribus uses an included hyphenation codebase as you've seen and clearly can produce different results to a python based hyphenator.

jirib

2025-01-09 05:58

reporter   ~0051893

Yes, the point to pyphen was wrong in detail but the attached C code produces "better" hyphenation, thus I think there's an area for improvement in Scribus - the attached code uses hnj_hyphen_hyphenate2().

And, there's hnj_hyphen_hyphenate3(), i'm not sure but seems like a new way to achieve hyphenation ???

cbradney

2025-01-09 21:28

administrator   ~0051903

Looking at the code at https://github.com/hunspell/hyphen/blob/master/hyphen.c, it looks like the 2nd function is the default. Our hyphen code has not been touched much in years. We'd need to see what we would need to change and understand how it would be different. It is something we could change in Scribus 1.7.0.

Issue History

Date Modified Username Field Change
2025-01-08 17:36 jirib New Issue
2025-01-08 18:57 jirib Note Added: 0051889
2025-01-08 19:15 jirib Note Added: 0051890
2025-01-08 19:15 jirib Note Added: 0051891
2025-01-08 19:15 jirib File Added: hyphen.c
2025-01-08 21:59 cbradney Note Added: 0051892
2025-01-09 05:58 jirib Note Added: 0051893
2025-01-09 21:28 cbradney Note Added: 0051903