|
|
Reminder sent to malex is this bug still there? |
|
|
Yes, Franz, the bug is still there. Just tested with newest cvs. Alex. |
|
2004-05-08 20:52
|
test_ru.png (125,251 bytes) |
|
2004-05-08 20:53
|
test_ru2.png (4,400 bytes) |
|
|
Hey, this one is funny: it seems that Scribus replaces on saving Russian diacritics in the file name with question marks (see attached files) |
|
|
Maciej, That happens, because you haven't built support for russian codepage such as koi8-r into your kernel. Look for it in "Filesystems>Native Language Support". |
|
|
Oleksandr, thanks for the hint. |
|
|
malex.. so, on your system is this now gone? |
|
|
Craig, The bug is still there. It is _not_ gone. |
|
|
The same happens with polish language in Scribus 1.1.7 :( It occurs when the file or directory contains polish diacritics... |
|
|
I paly with this [censored] locales. Looks like there is buggy Qt DOM parser... When I save scribus.rc file with UTF8 chars in the attribute I'll get string in uknown encoding (with ? instead utf8 chars). xml header: <?xml version='1.0' encoding="iso-88590-2"?> or: <?xml version='1.0' encoding="utf8"?> has no effect at all. That's strange because of the declared Utf8 text handling in Qt XML module... |
|
|
Reminder sent to fschmid see my bugnote, please |
|
|
Is this still kicking around ? |
|
|
We were corrupting 8-bit data in the prefs on every load/save. It worked for UTF-8 locales only because we were corrupting it twice in a reversable way. |
|
|
Regarding test_ru2.png, that's expected behaviour I'm afraid. Your computer is running in a locale that doesn't support the characters you're trying to use in the filename. You should be using a UTF-8 locale, or a suitable Russian locale, if you want to save filenames in russian. Most modern distros use a UTF-8 locale by default. Gentoo appears to be one exception, I'm sure there are others (Slackware?). Under debian you can change it using "dpkg-reconfigure locales" (if it's not installed, install it). You can often change your startup scripts to set your locale too; for example: LANG="en_AU.UTF-8" LC_ALL="en_AU.UTF-8" Note that this *WILL* cause filenames in the old locale to have 8-bit characters appear mangled. |
|
|
Re-broken by Franz'z diff, scribusXml.cpp 1.83.2.119 -> 1.83.2.120 . |
|
2005-07-08 03:33
|
broken.diff (1,111 bytes) |
|
|
Added detailed steps to reproduce. |
|
|
Craig, my correction and your diff affect only the contents of the file, not the filename handling. And if i would apply your latest diff the preferences file couldn't be read in again on my system. FYI .toCString() always returns an UTF8 encoded string, so there is no need to encode it twice. And this double applied encoding causes the ReadPrefs()function to fail. |
|
|
Here's an example out of the Qt Documentation: void ABCentralWidget::save( const QString &filename ) { if ( !listView->firstChild() ) return; QFile f( filename ); if ( !f.open( IO_WriteOnly ) ) return; QTextStream t( &f ); t.setEncoding(QTextStream::UnicodeUTF8); QListViewItemIterator it( listView ); for ( ; it.current(); ++it ) for ( unsigned int i = 0; i < 4; i++ ) t << it.current()->text( i ) << "\n"; f.close(); } void ABCentralWidget::load( const QString &filename ) { listView->clear(); QFile f( filename ); if ( !f.open( IO_ReadOnly ) ) return; QTextStream t( &f ); t.setEncoding(QTextStream::UnicodeUTF8); while ( !t.atEnd() ) { QListViewItem *item = new QListViewItem( listView ); for ( unsigned int i = 0; i < 4; i++ ) item->setText( i, t.readLine() ); } f.close(); } Note that there is no QString::fromUtf8() used at all, maybe that causes the corruption of the contents of the prefs file. |
|
|
QString QDomDocument::toString () const Converts the parsed document back to its textual representation. So, the code: QTextStream s(&f); s.setEncoding(QTextStream::UnicodeUTF8); s<<docu.toString(); - Creates a text stream - Tells the text stream convert QStrings etc in utf-8 format. Remember that QStrings are internally stored in UCS-2 . - Creates a UCS-2 encoded QString (all QStrings are UCS-2 encoded and must be explicitly re-coded on I/O) from the document - Writes the UCS-2 encoded QString to the text stream, which converts it to UTF-8 as it writes it out. I'm certain that's correct. It tests correct, the data is written out correctly, and I can see valid unicode data in the prefs file. When it comes to loading: QString encodedPrefs = QString::fromUtf8(f.readAll()); docu.setContent(encodedPrefs) Reads byte data from the file (which we know is UTF-8 encoded - we saved it that way) and loads it into a QString, telling the QString to treat the input bytes as UTF-8 when converting to its UCS-2 internal representation. It then passes QDomDocument a QString, which is correctly UCS-2 encoded, to parse. I'm also certain that is correct. |
|
|
Can I get you to post a copy of your prefs file so I can see what might be going on? |
|
|
As for the documentation snippet, you're quite right in that it would be fine to create a QTextStream from the QFile, set its encoding to UTF-8, then read its contents. If I'm right, the results should be exactly the same as what my patch did anyway. I'll test this soon, I'm still investigating what exactly is now going on. |
|
|
And here's why your changes were corrupting the prefs: QTextStream & QTextStream::operator<< ( const char * s ) ... The string s is assumed to be Latin1 encoded independent of the Encoding set for the QTextStream. In other words, the code: - Gets the QDomDocument text as a UTF-8 encoded QCString - Passes that UTF-8 encoded QCString to QTextStream::operator<<(...) - QTextStream takes what it thinks is latin-1 encoded text and *decodes* it as latin-1, then *encodes* it as UTF-8. Since it was already utf-8, this corrupts the text. |
|
|
I'd like to investigate this further. I really need a sample prefs file to do so, though, preferably one from before either you or I made any changes, one from when I made my changes, and one from now after both sets are in. |
|
|
As for reading, the docs don't say explictly what encoding QIODevice input to QDomDocument::setContent(...) is treated as having, but do say: ---------- bool QDomDocument::setContent ( const QCString & buffer, bool namespaceProcessing, QString * errorMsg = 0, int * errorLine = 0, int * errorColumn = 0 ) This is an overloaded member function, provided for convenience. It behaves essentially like the above function. This function reads the XML document from the C string buffer. ---------- and ---------- Warning: This function does not try to detect the encoding: instead it assumes that the C string is UTF-8 encoded. bool QDomDocument::setContent ( QIODevice * dev, bool namespaceProcessing, QString * errorMsg = 0, int * errorLine = 0, int * errorColumn = 0 ) This is an overloaded member function, provided for convenience. It behaves essentially like the above function. This function reads the XML document from the IO device dev. ---------- by which I'm assuming it'll treat the incoming text stream as UTF-8. Because of this lack of clarity I chose to read the stream myself and explicitly tell QString to treat the UTF-8 data as UTF-8. We could use QTextStream, too. |
|
|
>my correction and your diff affect only the contents of the file, not the filename handling. The contents of the file include the filenames for the recent docs menu, hence this doesnt work if the encoding is wrong. |
|
|
This discussion so far can thus be summed up as "encodings are a pain in the ass and really hard to get right". |
|
|
the main reason for my change was the fact that craig's correction makes it impossible to read back the prefs correctly here. So the correct solution is to fix reading the prefs again. |
|
|
OK, understood. To fix that, I'll need your prefs file from now, and from before any of these changes. One from just after my changes would also be very helpful. I know that's a pain, but I can't reproduce these issues here and so I need those files to have any hope of figuring out what went wrong on your system. If you could also paste the output of the "locale" command (run in a terminal) that'd be very handy. |
|
|
An old prefs file, from before any of the changes, is most important. |
|
|
Will mail you my current working scribus13.rc file this evening, have no access to my system right now. |
|
|
Thanks, Franz. I'm out of time now, but will check things out some more as soon as I can. No great hurry on the prefs, though I do think this needs fixing before 1.3.0 . |
|
|
Fixed in 1.3cvs. Please test. |
|
|
Pasted from the mail I sent out: I just checked in a different fix. I was eventually able to reproduce the old error - it looks like I was getting an incomplete file read or incomplete UTF-8 conversion from QString::fromUtf8(...) . Weird either way and needs more investigation. Switching to using a QTextStream and reading the whole thing into a QString with QTextStream::read() worked fine. Please test to make sure this works for you all. I had no problems here, but then, I didn't with the last one either (in no combination of locale settings etc). Note that this fix makes *no* effort to fix the old corrupt preferences. In most cases they should load OK, just with some garbage instead of non-ASCII characters. Please move your prefs aside if you have problems. If it's all OK, I'll do the same thing to 1.2.3cvs. The plan for 1.2.3 is to have it copy scribus.rc to scribus123.rc if scribus123.rc doesn't already exist - like we do for 1.2.x -> 1.3.x . It'll try to recover the corrupt scribus.rc by loading it using the old method. The new file will be saved in correct UTF-8 and once it exists, the old prefs will be ignored. That should let <1.2.3 and >=1.2.3 be used side-by-side and avoid the need to nuke user prefs during a stable series. It will mean that 1.3 will need to check for scribus123.rc and load that in preference to scribus.rc when importing 1.2.x prefs, but I think that'll be alright. Sound sensible? Testing reports appreciated. |
|
|
It's not fixed in 1.2.x yet but I'll split that into a separate bug. |
|
|
Testing of related 0002518 would be much appreciated. |
- Anonymous
