View Issue Details

IDProjectCategoryView StatusLast Update
0016100ScribusTypographypublic2020-08-11 20:56
Reporterftake Assigned Tojghali  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version1.5.6.svn 
Target VersionFixed in Version1.5.6.svn 
Summary0016100: No line wrap between a CJK character and a Latin character (after a CJK)
DescriptionIt seems that line wrap does not happens between a CJK character and a Latin character (only when a Latin character follows after a CJK character).

As far as I inspected by using debugger, `lineBreaks` provides correct places for line wrap and "s" of something (see the picture attached) have the `ScLayout_LineBoundary` flag.


Note that:
The code snippets using SpecialChars::CJK_NOBREAK_BEFORE and SpecialChars::CJK_NOBREAK_AFTER seems to be dead. I need more investigation libicu supports all the cases those snippets implemented.
TagsNo tags attached.
PatchNo

Activities

ftake

2020-04-28 09:41

reporter  

no-line-break-after-CJK.png (112,536 bytes)
no-line-break-after-CJK.png (112,536 bytes)
no-line-break-after-CJK.sla (15,578 bytes)

jghali

2020-04-29 13:46

administrator   ~0047580

The issue is at line 2418 of pageitem_textframe.cpp:
if (i != 0 && implicitBreak(itemText.text(glyphClusters[i - 1].lastChar()), itemText.text(current.glyphs[currentIndex].firstChar())))
The implicitBreak() function does not consider the cases of latin character following CJK character or CJK character following latin character.

A quick fix is to replace the condition by:
if (i != 0 && current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary))
However I'm not sure it will not break the handling of consecutive hyphens.

Quick related question : does SpecialChars::CJK_NOBREAK_AFTER needs to be considered in the case of a CJK character followed by a latin character? Same question for SpecialChars::CJK_NOBREAK_BEFORE : does SpecialChars::CJK_NOBREAK_BEFORE needs to be considered in the case of a CJK character preceded by a latin character.

jghali

2020-04-29 13:53

administrator   ~0047581

>> The code snippets using SpecialChars::CJK_NOBREAK_BEFORE and SpecialChars::CJK_NOBREAK_AFTER seems to be dead. I need more investigation libicu supports all the cases those snippets implemented.

Yes, I am also thinking that a least some of this code could be removed as we are now using icu for getting possible line boundaries.

ftake

2020-04-30 15:09

reporter   ~0047587

Oh, CJK line break rules implemented in textshaper.c and specialchars.c are overridden by pageitem_textframe.c

> A quick fix is to replace the condition by:
> if (i != 0 && current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary))

I have not fully understand yet but I guess we miss "else if" in the following if blocks:

```
// remember possible break
if (shapedText.haveMoreText(i + 1, glyphClusters))
{
    const GlyphCluster& nextCluster = glyphClusters[i + 1];
    if (nextCluster.hasFlag(ScLayout_LineBoundary))
    {
        if (!current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary)
            && !current.glyphs[currentIndex].hasFlag(ScLayout_HyphenationPossible)
            && (itemText.text(a) != '-')
            && (itemText.text(a) != SpecialChars::SHYPHEN))
        {
            current.rememberBreak(i, breakPos, style.rightMargin());
        }
    }
}
```

> does SpecialChars::CJK_NOBREAK_AFTER needs to be considered in the case of a CJK character followed by a latin character?

Yes. For example, in "「abcd」", "「" has CJK_NOBREAK_AFTER and no line break should happen between "「" and "abcd". Of course, libicu supports this kind of rules.

> Yes, I am also thinking that a least some of this code could be removed as we are now using icu for getting possible line boundaries.

I noticed a limitation of the current Scribus implimentation. libicu provides three line break strictness [1]. The original code in specialchars.c seems to be the strict mode. For example, 「っ」has NO_BREAK_AFTER but the normal mode of libicu returns that character is breakable.

For now, Scribus does not specify strictness, so the current mode seems to be sicked to the normal mode. Since the normal mode is suitable for some cases, it would be great if we can change the strictness from Scribus's character style.

[1] http://userguide.icu-project.org/boundaryanalysis#TOC-Line-Breaking-Strictness-a-CSS-Property

We can set strictness by the argument of Locale.

```
m_lineIterator = BreakIterator::createLineInstance(Locale("ja@lb=strict"), status);
```

jghali

2020-05-01 19:50

administrator   ~0047588

I have done more investigation and it seems the culprit may be in fact the following block of code:
if (nextCluster.hasFlag(ScLayout_LineBoundary))
{
    if (!current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary)
        && !current.glyphs[currentIndex].hasFlag(ScLayout_HyphenationPossible)
        && (itemText.text(a) != '-')
        && (itemText.text(a) != SpecialChars::SHYPHEN))
    {
        current.rememberBreak(i, breakPos, style.rightMargin());
    }
}

The two conditions explicitly forbid to remember a break opportunity if we have two successive break opportunities. Not sure why. However it is sure that this is not good for CJK and this is in fact what primarily cause this issue. Suppressing the condition related to ScLayout_LineBoundary in second if() *almost* fix it.

The second problem is that the block of code which test implicit chars afterwards remember break on a lower i - 1 value:
if (i != 0 && implicitBreak(itemText.text(glyphClusters[i - 1].lastChar()), itemText.text(current.glyphs[currentIndex].firstChar())))
{
    current.rememberBreak(i - 1, breakPos);
}

After the first block of code has been modified, it seems that this second block of code becomes unnecessary. So I'd just comment it out.

Patch attached. Please test.

16100_no_linebreak_between_cjk_and_latin_chars.patch (1,376 bytes)
Index: scribus/pageitem_textframe.cpp
===================================================================
--- scribus/pageitem_textframe.cpp	(revision 23676)
+++ scribus/pageitem_textframe.cpp	(working copy)
@@ -2170,8 +2170,8 @@
 				const GlyphCluster& nextCluster = glyphClusters[i + 1];
 				if (nextCluster.hasFlag(ScLayout_LineBoundary))
 				{
-					if (!current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary)
-						&& !current.glyphs[currentIndex].hasFlag(ScLayout_HyphenationPossible)
+					if (/*!current.glyphs[currentIndex].hasFlag(ScLayout_LineBoundary)
+						&&*/ !current.glyphs[currentIndex].hasFlag(ScLayout_HyphenationPossible)
 						&& (itemText.text(a) != '-')
 						&& (itemText.text(a) != SpecialChars::SHYPHEN))
 					{
@@ -2415,11 +2415,11 @@
 			if ((itemText.text(a) == SpecialChars::COLBREAK) && (m_columns > 1))
 				goNextColumn = true;
 
-			if (i != 0 && implicitBreak(itemText.text(glyphClusters[i - 1].lastChar()), itemText.text(current.glyphs[currentIndex].firstChar())))
+			/*if (i != 0 && implicitBreak(itemText.text(glyphClusters[i - 1].lastChar()), itemText.text(current.glyphs[currentIndex].firstChar())))
 			{
 //				qDebug() << "rememberBreak implicitbreak @" << i-1;
 				current.rememberBreak(i - 1, breakPos);
-			}
+			}*/
 			current.isEmpty = (i - current.lineData.firstCluster + 1) == 0;
 
 			if (tabs.active)

ftake

2020-05-04 02:04

reporter   ~0047590

Thank you. This patch works for me.

Issue History

Date Modified Username Field Change
2020-04-28 09:41 ftake New Issue
2020-04-28 09:41 ftake File Added: no-line-break-after-CJK.png
2020-04-28 09:41 ftake File Added: no-line-break-after-CJK.sla
2020-04-29 13:46 jghali Note Added: 0047580
2020-04-29 13:53 jghali Note Added: 0047581
2020-04-30 15:09 ftake Note Added: 0047587
2020-05-01 19:50 jghali File Added: 16100_no_linebreak_between_cjk_and_latin_chars.patch
2020-05-01 19:50 jghali Note Added: 0047588
2020-05-03 21:36 jghali Assigned To => jghali
2020-05-03 21:36 jghali Status new => resolved
2020-05-03 21:36 jghali Resolution open => fixed
2020-05-03 21:36 jghali Fixed in Version => 1.5.6.svn
2020-05-04 02:04 ftake Note Added: 0047590
2020-08-11 20:56 cbradney Status resolved => closed