View Issue Details

IDProjectCategoryView StatusLast Update
0000349ScribusUser Interfacepublic2005-09-08 08:55
Reportermalex Assigned Toringerc 
PrioritynormalSeverityblockReproducibilityalways
Status closedResolutionfixed 
Platformall 
Product Version1.2 
Fixed in Version1.3.0cvs 
Summary0000349: If a file is saved under a default name under a non-English locale it does not show up in the "Recent documents" submenu.
DescriptionThis is reproducible on my system when run as "scribus --lang uk | ru". If I create a new file and save it under the default name I can see the default file's translated name in the "Recent documents" _before_ I close the program. However, once I close Scribus and restart it, the last file will not show up in the "Recent documents" menu. It does not happen with the "--lang en", but happens consistently with "--lang uk | ru"
Steps To ReproduceFirst, set this page to display in UTF-8. In Firefox, View->Character Encoding->UTF-8. I have pasted text in a utf-8 encoding here, so that is required to see the page correctly.

SCRIBUS=/path/to/your/scribus/1.3/cvs

$ LANG=ru_RU LC_ALL=ru_RU $SCRIBUS
Create document
Save document with default name
Check recent scripts menu
Note doc is there in recent docs menu
Quit scribus

$ Edit scribus13.rc in a text editor that lets you confirm that the file is being viewed as utf-8 encoded. In vim, use :set fileencoding? to confirm.
Search for 'RECENT'.
Note the corrupt string for the recent document. It might look like this, though it'll depend a little on your editor:

<RECENT NAME="/home/craig/tmp/?<94>?????<83>???????<82>-2.sla" />

$ LANG=ru_RU LC_ALL=ru_RU $SCRIBUS
Note doc has vanished from recent scripts menu
Quit scribus

Also:
$ LANG=ru_RU.UTF-8 LC_ALL=ru_RU.UTF-8 $SCRIBUS
Create a new doc
Save as default name
Note doc is shown in recent scripts menu
Quit Scribus

$ Edit scribus13.rc in a text editor that lets you confirm that the file is being viewed as utf-8 encoded. In vim, use :set fileencoding? to confirm.
Search for 'RECENT'.
Note the differently corrupt string for the recent document. It'll might look like this, depending on your editor, fonts, etc:

<RECENT NAME="/home/craig/tmp/Ð<94>окÑ<83>менÑ<82>-1.sla" />

$ LANG=ru_RU.UTF-8 LC_ALL=ru_RU.UTF-8 $SCRIBUS
Note that the doc has vanished from the recent docs menu
Quit scribus
Additional Informationscribus13.rc should always be utf-8 encoded with correct, non-corrupt utf-8 text saved.
TagsNo tags attached.
Patch

Relationships

related to 0002120 closedringerc Some sort of jpeg images can't be loaded 
related to 0002518 closedringerc Prefs corruption in 1.2.x 
child of 0002215 closedjghali Audit and fix text encoding handling 

Activities

fschmid

2004-05-08 09:38

administrator   ~0001385

Reminder sent to malex

is this bug still there?

malex

2004-05-08 19:50

administrator   ~0001388

Yes, Franz, the bug is still there. Just tested with newest cvs.
Alex.

2004-05-08 20:52

 

test_ru.png (125,251 bytes)   
test_ru.png (125,251 bytes)   

2004-05-08 20:53

 

test_ru2.png (4,400 bytes)   
test_ru2.png (4,400 bytes)   

mhanski

2004-05-08 20:53

developer   ~0001391

Hey, this one is funny: it seems that Scribus replaces on saving Russian diacritics in the file name with question marks (see attached files)

malex

2004-05-10 14:37

administrator   ~0001413

Maciej,
That happens, because you haven't built support for russian codepage such as koi8-r into your kernel. Look for it in "Filesystems>Native Language Support".

mhanski

2004-05-10 20:21

developer   ~0001414

Oleksandr, thanks for the hint.

cbradney

2004-05-10 21:23

administrator   ~0001416

malex.. so, on your system is this now gone?

malex

2004-05-11 01:43

administrator   ~0001419

Craig,
The bug is still there. It is _not_ gone.

deejay1

2004-07-07 13:26

developer   ~0001923

The same happens with polish language in Scribus 1.1.7 :(
It occurs when the file or directory contains polish diacritics...

subik

2004-08-24 17:46

manager   ~0002214

I paly with this [censored] locales. Looks like there is buggy Qt DOM parser... When I save scribus.rc file with UTF8 chars in the attribute I'll get string in uknown encoding (with ? instead utf8 chars).
xml header:
<?xml version='1.0' encoding="iso-88590-2"?>
or:
<?xml version='1.0' encoding="utf8"?>
has no effect at all.

That's strange because of the declared Utf8 text handling in Qt XML module...

subik

2004-08-24 17:47

manager   ~0002215

Reminder sent to fschmid

see my bugnote, please

plinnell

2005-05-08 17:47

viewer   ~0004546

Is this still kicking around ?

ringerc

2005-07-02 08:41

reporter   ~0005330

We were corrupting 8-bit data in the prefs on every load/save. It worked for UTF-8 locales only because we were corrupting it twice in a reversable way.

ringerc

2005-07-02 08:43

reporter   ~0005331

Last edited: 2005-07-02 08:48

Regarding test_ru2.png, that's expected behaviour I'm afraid. Your computer is running in a locale that doesn't support the characters you're trying to use in the filename. You should be using a UTF-8 locale, or a suitable Russian locale, if you want to save filenames in russian.

Most modern distros use a UTF-8 locale by default. Gentoo appears to be one exception, I'm sure there are others (Slackware?). Under debian you can change it using "dpkg-reconfigure locales" (if it's not installed, install it).

You can often change your startup scripts to set your locale too; for example:

LANG="en_AU.UTF-8"
LC_ALL="en_AU.UTF-8"

Note that this *WILL* cause filenames in the old locale to have 8-bit characters appear mangled.

ringerc

2005-07-08 03:16

reporter   ~0005428

Re-broken by Franz'z diff, scribusXml.cpp 1.83.2.119 -> 1.83.2.120 .

2005-07-08 03:33

 

broken.diff (1,111 bytes)   
Index: scribusXml.cpp
===================================================================
RCS file: /cvs/Scribus/scribus/scribusXml.cpp,v
retrieving revision 1.83.2.119
retrieving revision 1.83.2.120
diff -u -r1.83.2.119 -r1.83.2.120
--- scribusXml.cpp	2 Jul 2005 11:06:05 -0000	1.83.2.119
+++ scribusXml.cpp	2 Jul 2005 14:30:18 -0000	1.83.2.120
@@ -3819,8 +3819,9 @@
 	if(!f.open(IO_WriteOnly))
 		return;
 	QTextStream s(&f);
-	s.setEncoding(QTextStream::UnicodeUTF8);
-	s<<docu.toString();
+	s<<docu.toCString();
+//	s.setEncoding(QTextStream::UnicodeUTF8);
+//	s<<docu.toString();
 	f.close();
 }
 
@@ -3830,13 +3831,19 @@
 	QFile f(ho);
 	if(!f.open(IO_ReadOnly))
 		return false;
-	QString encodedPrefs = QString::fromUtf8(f.readAll());
-	if( !docu.setContent(encodedPrefs) )
+	if(!docu.setContent(&f))
 	{
 		f.close();
 		return false;
 	}
 	f.close();
+/*	QString encodedPrefs = QString::fromUtf8(f.readAll());
+	if( !docu.setContent(encodedPrefs) )
+	{
+		f.close();
+		return false;
+	}
+	f.close(); */
 	QDomElement elem=docu.documentElement();
 	if (elem.tagName() != "SCRIBUSRC")
 		return false;
broken.diff (1,111 bytes)   

ringerc

2005-07-08 03:45

reporter   ~0005429

Added detailed steps to reproduce.

fschmid

2005-07-08 03:56

administrator   ~0005430

Craig, my correction and your diff affect only the contents of the file, not the filename handling. And if i would apply your latest diff the preferences file couldn't be read in again on my system. FYI .toCString() always returns an UTF8 encoded string, so there is no need to encode it twice. And this double applied encoding causes the ReadPrefs()function to fail.

fschmid

2005-07-08 04:18

administrator   ~0005431

Here's an example out of the Qt Documentation:

void ABCentralWidget::save( const QString &filename )
{
    if ( !listView->firstChild() )
        return;
    QFile f( filename );
    if ( !f.open( IO_WriteOnly ) )
        return;
    QTextStream t( &f );
    t.setEncoding(QTextStream::UnicodeUTF8);
    QListViewItemIterator it( listView );
    for ( ; it.current(); ++it )
        for ( unsigned int i = 0; i < 4; i++ )
            t << it.current()->text( i ) << "\n"; f.close();
}

void ABCentralWidget::load( const QString &filename )
{
    listView->clear();
    QFile f( filename );
    if ( !f.open( IO_ReadOnly ) )
        return;
    QTextStream t( &f );
    t.setEncoding(QTextStream::UnicodeUTF8);
    while ( !t.atEnd() )
    {
        QListViewItem *item = new QListViewItem( listView );
        for ( unsigned int i = 0; i < 4; i++ )
            item->setText( i, t.readLine() );
    }
    f.close();
}

Note that there is no QString::fromUtf8() used at all, maybe that causes the
corruption of the contents of the prefs file.

ringerc

2005-07-08 04:28

reporter   ~0005432

QString QDomDocument::toString () const
Converts the parsed document back to its textual representation.

So, the code:

QTextStream s(&f);
s.setEncoding(QTextStream::UnicodeUTF8);
s<<docu.toString();

- Creates a text stream
- Tells the text stream convert QStrings etc in utf-8 format. Remember that QStrings are internally stored in UCS-2 .
- Creates a UCS-2 encoded QString (all QStrings are UCS-2 encoded and must be explicitly re-coded on I/O) from the document
- Writes the UCS-2 encoded QString to the text stream, which converts it to UTF-8 as it writes it out.

I'm certain that's correct. It tests correct, the data is written out correctly, and I can see valid unicode data in the prefs file.

When it comes to loading:

QString encodedPrefs = QString::fromUtf8(f.readAll());
docu.setContent(encodedPrefs)

Reads byte data from the file (which we know is UTF-8 encoded - we saved it that way) and loads it into a QString, telling the QString to treat the input bytes as UTF-8 when converting to its UCS-2 internal representation. It then passes QDomDocument a QString, which is correctly UCS-2 encoded, to parse.

I'm also certain that is correct.

ringerc

2005-07-08 04:30

reporter   ~0005433

Can I get you to post a copy of your prefs file so I can see what might be going on?

ringerc

2005-07-08 04:35

reporter   ~0005434

As for the documentation snippet, you're quite right in that it would be fine to create a QTextStream from the QFile, set its encoding to UTF-8, then read its contents. If I'm right, the results should be exactly the same as what my patch did anyway. I'll test this soon, I'm still investigating what exactly is now going on.

ringerc

2005-07-08 04:41

reporter   ~0005435

And here's why your changes were corrupting the prefs:

QTextStream & QTextStream::operator<< ( const char * s )
...
The string s is assumed to be Latin1 encoded independent of the Encoding set for the QTextStream.

In other words, the code:

- Gets the QDomDocument text as a UTF-8 encoded QCString
- Passes that UTF-8 encoded QCString to QTextStream::operator<<(...)
- QTextStream takes what it thinks is latin-1 encoded text and *decodes* it as latin-1, then *encodes* it as UTF-8. Since it was already utf-8, this corrupts the text.

ringerc

2005-07-08 04:45

reporter   ~0005436

I'd like to investigate this further. I really need a sample prefs file to do so, though, preferably one from before either you or I made any changes, one from when I made my changes, and one from now after both sets are in.

ringerc

2005-07-08 04:49

reporter   ~0005437

As for reading, the docs don't say explictly what encoding QIODevice input to QDomDocument::setContent(...) is treated as having, but do say:

----------
bool QDomDocument::setContent ( const QCString & buffer, bool namespaceProcessing, QString * errorMsg = 0, int * errorLine = 0, int * errorColumn = 0 )
This is an overloaded member function, provided for convenience. It behaves essentially like the above function.

This function reads the XML document from the C string buffer.
----------

and

----------
Warning: This function does not try to detect the encoding: instead it assumes that the C string is UTF-8 encoded.

bool QDomDocument::setContent ( QIODevice * dev, bool namespaceProcessing, QString * errorMsg = 0, int * errorLine = 0, int * errorColumn = 0 )
This is an overloaded member function, provided for convenience. It behaves essentially like the above function.

This function reads the XML document from the IO device dev.
----------

by which I'm assuming it'll treat the incoming text stream as UTF-8. Because of this lack of clarity I chose to read the stream myself and explicitly tell QString to treat the UTF-8 data as UTF-8. We could use QTextStream, too.

cbradney

2005-07-08 05:21

administrator   ~0005438

>my correction and your diff affect only the contents of the file, not the filename handling.

The contents of the file include the filenames for the recent docs menu, hence this doesnt work if the encoding is wrong.

ringerc

2005-07-08 05:27

reporter   ~0005439

This discussion so far can thus be summed up as "encodings are a pain in the ass and really hard to get right".

fschmid

2005-07-08 05:52

administrator   ~0005440

the main reason for my change was the fact that craig's correction makes it
impossible to read back the prefs correctly here. So the correct solution is
to fix reading the prefs again.

ringerc

2005-07-08 06:00

reporter   ~0005441

OK, understood.

To fix that, I'll need your prefs file from now, and from before any of these changes. One from just after my changes would also be very helpful. I know that's a pain, but I can't reproduce these issues here and so I need those files to have any hope of figuring out what went wrong on your system.

If you could also paste the output of the "locale" command (run in a terminal) that'd be very handy.

ringerc

2005-07-08 06:16

reporter   ~0005442

An old prefs file, from before any of the changes, is most important.

fschmid

2005-07-08 06:31

administrator   ~0005443

Will mail you my current working scribus13.rc file this evening, have no access to my system right now.

ringerc

2005-07-08 07:31

reporter   ~0005444

Thanks, Franz.

I'm out of time now, but will check things out some more as soon as I can. No great hurry on the prefs, though I do think this needs fixing before 1.3.0 .

ringerc

2005-07-09 04:22

reporter   ~0005459

Fixed in 1.3cvs. Please test.

ringerc

2005-07-09 04:24

reporter   ~0005460

Pasted from the mail I sent out:

I just checked in a different fix. I was eventually able to reproduce
the old error - it looks like I was getting an incomplete file read or
incomplete UTF-8 conversion from QString::fromUtf8(...) . Weird either
way and needs more investigation. Switching to using a QTextStream and
reading the whole thing into a QString with QTextStream::read() worked
fine.

Please test to make sure this works for you all. I had no problems here,
but then, I didn't with the last one either (in no combination of locale
settings etc).

Note that this fix makes *no* effort to fix the old corrupt preferences.
In most cases they should load OK, just with some garbage instead of
non-ASCII characters. Please move your prefs aside if you have problems.

If it's all OK, I'll do the same thing to 1.2.3cvs. The plan for 1.2.3
is to have it copy scribus.rc to scribus123.rc if scribus123.rc doesn't
already exist - like we do for 1.2.x -> 1.3.x . It'll try to recover the
corrupt scribus.rc by loading it using the old method. The new file will
be saved in correct UTF-8 and once it exists, the old prefs will be
ignored.

That should let <1.2.3 and >=1.2.3 be used side-by-side and avoid the
need to nuke user prefs during a stable series. It will mean that 1.3
will need to check for scribus123.rc and load that in preference to
scribus.rc when importing 1.2.x prefs, but I think that'll be alright.

Sound sensible?

Testing reports appreciated.

ringerc

2005-07-09 09:13

reporter   ~0005467

It's not fixed in 1.2.x yet but I'll split that into a separate bug.

ringerc

2005-09-02 10:57

reporter   ~0006346

Testing of related 0002518 would be much appreciated.

Issue History

Date Modified Username Field Change
2004-03-15 21:51 malex New Issue
2004-04-07 14:44 cbradney Category General => User Interface
2004-05-08 09:38 fschmid Note Added: 0001385
2004-05-08 19:50 malex Note Added: 0001388
2004-05-08 20:52 mhanski File Added: test_ru.png
2004-05-08 20:53 mhanski File Added: test_ru2.png
2004-05-08 20:53 mhanski Note Added: 0001391
2004-05-10 14:37 malex Note Added: 0001413
2004-05-10 20:21 mhanski Note Added: 0001414
2004-05-10 21:23 cbradney Note Added: 0001416
2004-05-11 01:43 malex Note Added: 0001419
2004-07-07 13:26 deejay1 Note Added: 0001923
2004-08-24 17:46 subik Note Added: 0002214
2004-08-24 17:47 subik Note Added: 0002215
2005-05-08 17:47 plinnell Note Added: 0004546
2005-07-02 07:31 ringerc Status new => assigned
2005-07-02 07:31 ringerc Assigned To => ringerc
2005-07-02 07:56 ringerc Relationship added related to 0002120
2005-07-02 08:41 ringerc Status assigned => resolved
2005-07-02 08:41 ringerc Fixed in Version => 1.3.0cvs
2005-07-02 08:41 ringerc Resolution open => fixed
2005-07-02 08:41 ringerc Note Added: 0005330
2005-07-02 08:43 ringerc Note Added: 0005331
2005-07-02 08:48 ringerc Note Edited: 0005331
2005-07-03 09:17 cbradney Status resolved => closed
2005-07-08 03:16 ringerc Status closed => feedback
2005-07-08 03:16 ringerc Resolution fixed => reopened
2005-07-08 03:16 ringerc Note Added: 0005428
2005-07-08 03:16 ringerc Status feedback => confirmed
2005-07-08 03:30 ringerc Steps to Reproduce Updated
2005-07-08 03:32 ringerc Steps to Reproduce Updated
2005-07-08 03:32 ringerc Additional Information Updated
2005-07-08 03:33 ringerc File Added: broken.diff
2005-07-08 03:43 ringerc Steps to Reproduce Updated
2005-07-08 03:45 ringerc Steps to Reproduce Updated
2005-07-08 03:45 ringerc Note Added: 0005429
2005-07-08 03:56 fschmid Note Added: 0005430
2005-07-08 04:18 fschmid Note Added: 0005431
2005-07-08 04:28 ringerc Note Added: 0005432
2005-07-08 04:30 ringerc Note Added: 0005433
2005-07-08 04:35 ringerc Note Added: 0005434
2005-07-08 04:41 ringerc Note Added: 0005435
2005-07-08 04:45 ringerc Note Added: 0005436
2005-07-08 04:49 ringerc Note Added: 0005437
2005-07-08 05:21 cbradney Note Added: 0005438
2005-07-08 05:27 ringerc Note Added: 0005439
2005-07-08 05:52 fschmid Note Added: 0005440
2005-07-08 06:00 ringerc Note Added: 0005441
2005-07-08 06:16 ringerc Note Added: 0005442
2005-07-08 06:31 fschmid Note Added: 0005443
2005-07-08 07:31 ringerc Note Added: 0005444
2005-07-08 07:51 cbradney Severity minor => block
2005-07-08 08:01 cbradney Note View State: 5432: private
2005-07-08 08:01 cbradney Note View State: 5432: public
2005-07-08 08:16 ringerc Status confirmed => assigned
2005-07-09 04:22 ringerc Note Added: 0005459
2005-07-09 04:24 ringerc Note Added: 0005460
2005-07-09 08:45 fschmid Status assigned => resolved
2005-07-09 08:45 fschmid Resolution reopened => fixed
2005-07-09 09:13 ringerc Note Added: 0005467
2005-07-10 03:01 ringerc Relationship added child of 0002215
2005-09-02 09:57 ringerc Relationship added related to 0002518
2005-09-02 10:57 ringerc Note Added: 0006346
2005-09-08 08:55 ringerc Status resolved => closed