View Issue Details

IDProjectCategoryView StatusLast Update
0010292ScribusStory Editor / Text Framespublic2016-04-11 00:03
ReporterSherwood Wang Assigned Tojghali  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Platformx86_64OSDebianOS VersionSqueeze
Product Version1.5.0svn 
Fixed in Version1.5.0svn 
Summary0010292: [PATCH] CJK formating issue
DescriptionPeople who write in CJK don't add spaces between characters. So Scribus will regard a whole paragraph of CJK characters as one word. It will not break lines between two CJK characters if there is any space at other place. And it can't justify such a line for the same reason.
This patch will identify CJK characters and add implicit breakable points between them to correct line-breaking problem add implicit spaces to correct line-justifying problem.
Steps To ReproduceJust copy any Chinese text (especially with some spaces in it) into the text frame.
TagsNo tags attached.
Patch

Activities

Sherwood Wang

2011-10-01 02:56

reporter  

cjkpatch (5,419 bytes)   
Index: scribus/pageitem_textframe.cpp
===================================================================
--- scribus/pageitem_textframe.cpp	(revision 16866)
+++ scribus/pageitem_textframe.cpp	(working copy)
@@ -724,8 +724,68 @@
 	double lineCorr;
 };
 
+static int checkCJK(QChar ch) {
+	unsigned int code = ch.unicode();
+	if (	(0x2E80 < code && code < 0x2EFF) ||   // CJK Radicals Supplement
+		(0x3000 < code && code < 0x303F) ||   // CJK Symbols and Punctuation
+		(0x31C0 < code && code < 0x31EF) ||   // CJK Strokes
+		(0x3200 < code && code < 0x32FF) ||   // Enclosed CJK Letters and Months
+		(0x3300 < code && code < 0x33FF) ||   // CJK Compatibility
+		(0x3400 < code && code < 0x4DBF) ||   // CJK Unified Ideographs Extension A
+		(0x4E00 < code && code < 0x9FFF) ||   // CJK Unified Ideographs
+		(0xF900 < code && code < 0xFAFF) ||   // CJK Compatibility Ideographs
+		(0xFE30 < code && code < 0xFE4F) ||   // CJK Compatibility Forms
+		(0x20000 < code && code < 0x2A6DF) || // CJK Unified Ideographs Extension B
+		(0x2A700 < code && code < 0x2B73F) || // CJK Unified Ideographs Extension C
+		(0x2B740 < code && code < 0x2B81F) || // CJK Unified Ideographs Extension D
+		(0x2F800 < code && code < 0x2FA1F) || // CJK Compatibility Ideographs Supplement
+		(0xFF01 < code && code < 0xFF0F) ||
+		(0xFF1A < code && code < 0xFF20) ||
+		(0xFF58 < code && code < 0xFFDC) ||
+		(code == 0x3000) ||
+		(code == 0x3002) ||
+		(code == 0x201C) ||
+		(code == 0x201D))
+		return 1;
+	else
+		return 0;
+}
 
+static int checkCJKBreakAfter(QChar ch) {
+	unsigned int code[] = {0x201C, 0xFF08, 0xFF3B, 0xFF5B, 0xFF5F, 0xFF62, 0xFF0D, 0};
+	for (int i = 0; code[i]; ++i)
+		if (code[i] == ch.unicode())
+			return 0;
+	return 1;
+}
 
+static int checkCJKBreakBefore(QChar ch) {
+	unsigned int code[] =
+	 {0x201D, 0x3002, 0xFF01, 0xFF09, 0xFF0C, 0xFF0E, 0xFF1A,
+	  0xFF1B, 0xFF1F, 0xFF3D, 0xFF5D, 0xFF60, 0xFF63, 0xFF64, 0};
+	for (int i = 0; code[i]; ++i)
+		if (code[i] == ch.unicode())
+			return 0;
+	return 1;
+}
+
+static int implicitSpace(ScText *f, ScText *s) {
+	if (checkCJK(f->ch) && checkCJK(s->ch))
+		return 1;
+	else
+		return 0;
+}
+
+static int implicitBreak(ScText *f, ScText *s) {
+	if (checkCJK(f->ch) && checkCJK(s->ch)) {
+		if (!checkCJKBreakAfter(f->ch)
+		 || !checkCJKBreakBefore(s->ch))
+			return 0;
+		return 1;
+	} else
+		return 0;
+}
+
 /// called when line length is known and line is to be justified
 static void justifyLine(StoryText& itemText, LineSpec& line)
 {
@@ -734,6 +794,8 @@
 	double spaceNatural = 0;
 	double glyphExtension;
 	double spaceExtension;
+	int spaceInsertion = 0;
+	double imSpace = -1;
 
 	const ParagraphStyle& style(itemText.paragraphStyle(line.firstItem));
 
@@ -747,28 +809,53 @@
 		else if ( (itemText.item(sof)->effects() & ScStyle_SuppressSpace) == 0)
 		{
 			spaceNatural += itemText.item(sof)->glyph.wide();
+			if (imSpace < 0.0 || imSpace > itemText.item(sof)->glyph.wide())
+				imSpace = itemText.item(sof)->glyph.wide();
 		}
+		if (sof != line.firstItem && implicitSpace(itemText.item(sof - 1), itemText.item(sof))) {
+			spaceInsertion += 1;
+		}
 	}
 
-	// decision: prio 1: stretch glyph;  prio 2: stretch spaces
+	imSpace /= 2;
+
+	// decision: prio 1: stretch glyph;  prio 2: insert spaces;  prio 3: stretch spaces
 	
 	if (line.width < spaceNatural + glyphNatural * style.minGlyphExtension() && spaceNatural > 0)
 	{
 		glyphExtension = style.minGlyphExtension() - 1;
 		spaceExtension = (line.width - glyphNatural * (1+glyphExtension) ) / spaceNatural  - 1;
+		imSpace = 0;
 	}
 	else if (line.width < spaceNatural + glyphNatural * style.maxGlyphExtension() && glyphNatural > 0)
 	{
 		spaceExtension = 0;
 		glyphExtension = (line.width - spaceNatural) / glyphNatural  - 1;
+		imSpace = 0;
 	}
 	else
 	{
 		glyphExtension = style.maxGlyphExtension() - 1;
-		if (spaceNatural > 0)
-			spaceExtension = (line.width - glyphNatural * (1+glyphExtension) ) / spaceNatural  - 1;
-		else
-			spaceExtension = 0;
+		if (spaceInsertion) {
+			double remaining = line.width - glyphNatural * (1 + glyphExtension) - spaceNatural;
+			if (imSpace > 0) {
+				if (remaining / spaceInsertion < imSpace) {
+					imSpace = remaining / spaceInsertion;
+					spaceExtension = 0;
+				} else {
+					spaceExtension = (remaining + spaceNatural) / (spaceNatural + spaceInsertion * imSpace) - 1;
+					imSpace *= spaceExtension + 1;
+				}
+			} else {
+				imSpace = remaining / spaceInsertion;
+				spaceExtension = 0;
+			}
+		} else {
+			if (spaceNatural > 0)
+				spaceExtension = (line.width - glyphNatural * (1+glyphExtension) ) / spaceNatural  - 1;
+			else
+				spaceExtension = 0;
+		}
 	}
 	
 	double glyphScale = 1 + glyphExtension;
@@ -799,6 +886,9 @@
 		{
 			itemText.item(yof)->glyph.last()->xadvance += wide * spaceExtension;
 		}
+		if (yof != line.firstItem && implicitSpace(itemText.item(yof - 1), itemText.item(yof))) {
+			itemText.item(yof - 1)->glyph.last()->xadvance += imSpace;
+		}
 	}
 }
 
@@ -1651,6 +1741,10 @@
 			if ((hl->ch == SpecialChars::COLBREAK) && (Cols > 1))
 				goNextColumn = true;
 
+			if (a != firstInFrame() && implicitBreak(itemText.item(a - 1), itemText.item(a))) {
+				current.rememberBreak(a - 1, breakPos);
+			}
+
 			// remember possible break
 			// #5783 : comment out "&& !outs" as this make it possible to perform a line break
 			// before a space which contradicts unicode line breaking rules
cjkpatch (5,419 bytes)   

ale

2011-10-02 07:11

manager   ~0026945

jiero tested it and said that it is really useful. (and works)

the patch looks very straightforward and clean. still, i'm not sure it's a good idea to put such language specific exceptions in the middle of pagitem_textframe... is there any alternative?

what i also wonder is: can full OTF support will help "cleanly" solving this issue? ... just wondering...

Sherwood Wang

2011-10-03 07:40

reporter   ~0026948

Last edited: 2011-10-03 07:40

I think it's inevitable to put some language specific codes into pagitem_textframe. In Latin, a character is a letter. In CJK, a character can be a word. It's almost impossible to layout CJK scripts like layouting Latin scripts. So XeTeX, which is developed by SIL International, adopted a similar method to break lines and add glues with CJK.
I have no idea about solving this issue with OTF. As I know, OTF doesn't define anything about lines. It only interests in the relation between characters.

ale

2011-10-03 13:39

manager   ~0026950

ok, if xatex does it that way, it must be the good way!

could you just check if other asiatic languages need a similar check (but with other characters) and eventually implement it, too?

we don't have any chinese devoper (yet!) but we would really like to be able to support asiatic languages!

Sherwood Wang

2011-10-04 17:37

reporter   ~0026967

Chinese (including some dialect of it, such as Yue, Kejiahua), Japanese and Korean are similar. Thanks to Unicode, all characters of this style are put into CJK sections, even some special characters are only used in the derivatives of these languages. So problems about them are solved by this patch.
I'm interested in Scribus. And I will try to make contributions.

jghali

2011-10-04 20:54

administrator   ~0026971

Committed! Many thanks!

Kunda

2016-04-11 00:03

updater   ~0040035

@Sherwood Wang:
we need testers and test files for ScribusCTL that could support Vertical Text: https://github.com/HOST-Oman/scribus/issues/143

Issue History

Date Modified Username Field Change
2011-10-01 02:56 Sherwood Wang New Issue
2011-10-01 02:56 Sherwood Wang File Added: cjkpatch
2011-10-02 07:11 ale Note Added: 0026945
2011-10-03 07:40 Sherwood Wang Note Added: 0026948
2011-10-03 07:40 Sherwood Wang Note Edited: 0026948
2011-10-03 13:39 ale Note Added: 0026950
2011-10-04 17:37 Sherwood Wang Note Added: 0026967
2011-10-04 20:54 jghali Note Added: 0026971
2011-10-04 20:54 jghali Status new => resolved
2011-10-04 20:54 jghali Fixed in Version => 1.5.0svn
2011-10-04 20:54 jghali Resolution open => fixed
2011-10-04 20:54 jghali Assigned To => jghali
2011-10-05 20:04 cbradney Status resolved => closed
2015-09-17 20:08 Kunda Category Story Editor / Text Frames => Story Ed/Txt Frames
2015-09-17 20:12 Kunda Category Story Ed/Txt Frames => Story Editor / Text Frames
2016-04-11 00:03 Kunda Note Added: 0040035