View Issue Details

IDProjectCategoryView StatusLast Update
0017367ScribusTypographypublic2025-01-09 05:58
Reporterjirib Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version1.6.3.svn 
Summary0017367: [hyphenator.cpp] hyphenation works differently even when using same hyphenation dict
DescriptionThis is how is the Czech word 'orgánu' hyphenated in Scribus:

$
$ grep -Po ' \K(or.*nu)\s+' pl-reprint-01.sla | xxd
00000000: 6f72 c2ad 67c3 a16e 7520 0a or..g..nu .

# in literal: or-gánu ('-' is soft hyphen)

A proof that same hyphen dict is used:

$ md5sum ~/.local/share/scribus/dicts/hyph/hyph_cs_CZ.dic /home/jiri/.venv3/lib/python3.11/site-packages/pyphen/dictionaries/hyph_cs_CZ.dic
c9cbd906b7884d88b00a3a0a6a7773c7 /home/jiri/.local/share/scribus/dicts/hyph/hyph_cs_CZ.dic
c9cbd906b7884d88b00a3a0a6a7773c7 /home/jiri/.venv3/lib/python3.11/site-packages/pyphen/dictionaries/hyph_cs_CZ.dic

With pyphen:

>>> import pyphen
>>> dic = pyphen.Pyphen(lang='cs_CZ')
>>> text = 'orgánu'
>>> ' '.join(dic.inserted(word) for word in text.split())
'or-gá-nu'

There's a TODO note in the code regarding 'hnj_hyphen_hyphenate2()' - https://github.com/scribusproject/scribus/blob/master/scribus/hyphenator.cpp#L116 .

Is that the reason?

The "impact" is unpleasant rivers of white :-(
Steps To ReproduceSee example above.
TagsNo tags attached.
PatchNo

Activities

jirib

2025-01-08 18:57

reporter   ~0051889

Eh, pyphen is python-only implementation, it does not use hunspell library. So, I don't know if this is a bug or not :/

jirib

2025-01-08 19:15

reporter   ~0051890

s/hunspell/hyphen/ - i'm very sorry, i'm new to this.

a test via C code from hyphen:

$ gcc -o hyphen_test /tmp/hyphen.c -lhyphen
$ ./hyphen_test
Hyphenation points for 'orgánu':
Hyphenation point at index 1
Hyphenation point at index 4
Hyphenated word: or-gá-nu

jirib

2025-01-08 19:15

reporter   ~0051891

C code for hyphen_test
hyphen.c (1,361 bytes)   
#include <stdio.h>
#include <string.h>
#include "hyphen.h"

int main() {
    HyphenDict *hd = hnj_hyphen_load("hyph_cs_CZ.dic");
    if (!hd) {
        fprintf(stderr, "Failed to load hyphenation patterns\n");
        return 1;
    }

    const char *word = "orgánu";
    int word_size = strlen(word);

    char hyphens[word_size + 5];
    memset(hyphens, 0, sizeof(hyphens));

    char hyphenated_word[word_size * 2]; 
    memset(hyphenated_word, 0, sizeof(hyphenated_word));

    char **rep = NULL; 
    int *pos = NULL;   
    int *cut = NULL;   

    int result = hnj_hyphen_hyphenate2(hd, word, word_size, hyphens,
                                       hyphenated_word, &rep, &pos, &cut);

    if (result != 0) {
        fprintf(stderr, "Hyphenation failed\n");
        hnj_hyphen_free(hd);
        return 1;
    }

    int index = 0;
    for (int i = 0; i < word_size; i++) {
        hyphenated_word[index++] = word[i];
        if (hyphens[i] % 2 != 0) { 
            hyphenated_word[index++] = '-';
        }
    }
    hyphenated_word[index] = '\0'; 

    printf("Hyphenation points for '%s':\n", word);
    for (int i = 0; i < word_size; i++) {
        if (hyphens[i] % 2 != 0) {
            printf("Hyphenation point at index %d\n", i);
        }
    }
    printf("Hyphenated word: %s\n", hyphenated_word);

    hnj_hyphen_free(hd); 
    return 0;
}
hyphen.c (1,361 bytes)   

cbradney

2025-01-08 21:59

administrator   ~0051892

Scribus uses an included hyphenation codebase as you've seen and clearly can produce different results to a python based hyphenator.

jirib

2025-01-09 05:58

reporter   ~0051893

Yes, the point to pyphen was wrong in detail but the attached C code produces "better" hyphenation, thus I think there's an area for improvement in Scribus - the attached code uses hnj_hyphen_hyphenate2().

And, there's hnj_hyphen_hyphenate3(), i'm not sure but seems like a new way to achieve hyphenation ???

Issue History

Date Modified Username Field Change
2025-01-08 17:36 jirib New Issue
2025-01-08 18:57 jirib Note Added: 0051889
2025-01-08 19:15 jirib Note Added: 0051890
2025-01-08 19:15 jirib Note Added: 0051891
2025-01-08 19:15 jirib File Added: hyphen.c
2025-01-08 21:59 cbradney Note Added: 0051892
2025-01-09 05:58 jirib Note Added: 0051893