View Issue Details

IDProjectCategoryView StatusLast Update
0015350ScribusImport / Exportpublic2019-06-04 20:56
ReporterralfD Assigned Tojghali  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version1.5.4.svn 
Fixed in Version1.5.5.svn 
Summary0015350: xtg import broken for multibyte encoded input files
Descriptionxtg import for multibyte encoded input files (UTF-8, UTF-16 etc.) is broken. The function XtgScanner::nextSymbol in file xtgscanner.cpp read characters and feeds them to m_codec->toUnicode which will fail for all mutibyte input.
Steps To ReproduceSave the following text to an (UTF-8 encoded) file:

x------------------------------[file: sample.xtg]------------------------------------------------------x
<v8.00><e9>
@mystyle:Fürchtet euch nicht, gössere Kläffer!
x----------------------------------------------------------------------------------------------------------x

Create a text frame and import the file as a 'Quark Xpress Tags Document'.

Additional InformationThe fix for this would be to switch to Qt's stateful decoder. In a quick/hacky test that does fix the issue.
TagsNo tags attached.
PatchNo

Activities

jghali

2018-06-05 18:45

administrator   ~0045302

Can you please provide a sample xtg file?

ralfD

2018-06-05 19:45

reporter   ~0045303

Yes, see the following minimal example.
multibyte.xtg (44 bytes)   
<v8.00><e9>
Können Küsse Käse schließen
multibyte.xtg (44 bytes)   

ralfD

2018-06-06 08:54

reporter   ~0045306

Just to note: lookAhead() suffers from the same shotcomings. To really fix this problem the scanner should operate on characters/on a character buffer and not on bytes/a bytearray.

jghali

2018-06-06 10:18

administrator   ~0045307

>> To really fix this problem the scanner should operate on characters/on a character buffer and not on bytes/a bytearray.

Yes, that's the easy solution. However the xtg format makes it theoretically possible to switch between encodings inside a single file. The xtg documentation is not clear about that. So I'm not quite sure that we can use a character buffer.

ralfD

2018-06-06 10:40

reporter   ~0045308

Yes, I was wondering about that myself. The "specs" are rather unspecific about where the encoding tag is allowed (and doesn't even mention compinations of BOM and encoding tag). The whole format seems to be rather ad-hoc (at least when I last looked at it while writing a parser almost 20 years ago ...).
But realistically I never encountered a file with more then one encoding marker (n.b.: all those files where exports from Quark XPress or related products).

Issue History

Date Modified Username Field Change
2018-06-05 15:41 ralfD New Issue
2018-06-05 18:45 jghali Note Added: 0045302
2018-06-05 19:45 ralfD File Added: multibyte.xtg
2018-06-05 19:45 ralfD Note Added: 0045303
2018-06-06 08:54 ralfD Note Added: 0045306
2018-06-06 10:18 jghali Note Added: 0045307
2018-06-06 10:40 ralfD Note Added: 0045308
2018-06-17 23:17 jghali Assigned To => jghali
2018-06-17 23:17 jghali Status new => resolved
2018-06-17 23:17 jghali Resolution open => fixed
2018-06-17 23:17 jghali Fixed in Version => 1.5.5.svn
2019-06-04 20:56 cbradney Status resolved => closed