View Issue Details

IDProjectCategoryView StatusLast Update
0008157ScribusLanguage Toolspublic2014-10-24 22:57
Reportercaolan Assigned Tocbradney  
PrioritynormalSeverityfeatureReproducibilityalways
Status closedResolutionwon't fix 
Product Version1.3.5svn 
Summary0008157: [PATCH] Support for arbitrary seperately packaged hyphenators
DescriptionScribus includes a copy of a number of hyphenators that were originally created
for use by OOo.

But there are a lot more hyphenators for a lot more languages available, and
e.g. they are generally now packaged separately in Fedora and Debian (see /usr/share/hyphen and hyphen-af through to hyphen-zu at
http://fedoraproject.org/wiki/OpenOffice.org/LinguisticComponents)

It would be ideal for those distros to be use the system hyphenators to reduce duplication and to allow scribus to support these other languages.

A simple link from /usr/share/scribus/dicts to /usr/share/hyphen shows a few problems with that "simple" approach.

Attached is a patch that allows this to be at least possible.
Additional Informationa) scribus/scribus.cpp
There's probably no reason that not process links within the /usr/lib/scribus/dicts dir itself, so remove | QDir::NoSymLinks

b) scribus/hyphenator.cpp
Make this code safe against, e.g. releasing a NULL hyphenator or attempts to load a hyphenator that doesn't exist

c) scribus/hysettings.cpp
The names of languages that scribus knows about are in a hard-coded table. So to support arbitrary languages which scribus doesn't know about we take the locale string as a fallback in the UI

d) scribus/langmgr.cpp
This bit is admittedly a bit messy given the existing table of language abbreviations that are in use. But the patch should be good
enough for the majority of situations as a starting point.

(as an aside, there are some odd entries in LanguageManager::generateLangList, e.g.
langList.insert("ga", langPair("Galician",
as "ga" is the language code for Irish (Gaelic), and
langList.insert("es_LA", langPair("Spanish (Latin)",
which suggest Spanish as spoken in Lao by its name)
TagsNo tags attached.
PatchYes

Activities

2009-06-15 13:20

 

scribus-1.3.5.rc2.arbitraryhyphenators.patch (5,009 bytes)   
diff -ru scribus-1.3.5.rc2.orig/scribus/scribus.cpp scribus-1.3.5.rc2/scribus/scribus.cpp
--- scribus-1.3.5.rc2.orig/scribus/scribus.cpp	2009-06-15 10:15:07.000000000 +0100
+++ scribus-1.3.5.rc2/scribus/scribus.cpp	2009-06-15 10:15:54.000000000 +0100
@@ -8841,7 +8841,7 @@
 	//Grab the language abbreviation from it, get the full language text
 	//Insert the name as key and a new string list into the map
 	QString hyphDirName = QDir::convertSeparators(ScPaths::instance().dictDir());
-	QDir hyphDir(hyphDirName, "*.dic", QDir::Name, QDir::Files | QDir::NoSymLinks);
+	QDir hyphDir(hyphDirName, "*.dic", QDir::Name, QDir::Files);
 	if ((hyphDir.exists()) && (hyphDir.count() != 0))
 	{
 // 		LanguageManager langmgr;
diff -ru scribus-1.3.5.rc2.orig/scribus/hyphenator.cpp scribus-1.3.5.rc2/scribus/hyphenator.cpp
--- scribus-1.3.5.rc2.orig/scribus/hyphenator.cpp	2009-06-15 10:15:19.000000000 +0100
+++ scribus-1.3.5.rc2/scribus/hyphenator.cpp	2009-06-15 12:49:16.000000000 +0100
@@ -61,23 +61,26 @@
 		Language = PrefsManager::instance()->appPrefs.Language;
 		doc->Language = Language;
 	}
+
 // 	pfad += ScCore->primaryMainWindow()->Sprachen[Language];
-	pfad += lmgr->getHyphFilename( doc->Language, false );
-	QFile f(pfad);
-	if (f.open(QIODevice::ReadOnly))
+	QString fileName = lmgr->getHyphFilename( doc->Language, false );
+ 	if (!fileName.isEmpty())
 	{
-		QTextStream st(&f);
-    	QString line;
-    	line = st.readLine();
-		codec = QTextCodec::codecForName(line.toUtf8());
-		f.close();
+		pfad += fileName;
+		QFile f(pfad);
+		if (f.open(QIODevice::ReadOnly))
+		{
+			QTextStream st(&f);
+			QString line;
+			line = st.readLine();
+			f.close();
+			useAble = true;
+		}
 	}
-	else
-	{
-		useAble = false;
-		hdict = NULL;
+
+	if (!useAble)
 		return;
-	}
+
 	QByteArray fn = pfad.toLocal8Bit();
 	const char * filename = fn.data();
 	hdict = hnj_hyphen_load(filename);
@@ -90,7 +93,8 @@
 
 Hyphenator::~Hyphenator()
 {
-	hnj_hyphen_free(hdict);
+	if (hdict != NULL)
+		hnj_hyphen_free(hdict);
 }
 
 void Hyphenator::NewDict(const QString& name)
@@ -110,22 +114,29 @@
 		if (hdict != NULL)
 			hnj_hyphen_free(hdict);
 
-		pfad +=  LanguageManager::instance()->getHyphFilename(Language, false) ;
-		QFile f(pfad);
-		if (f.open(QIODevice::ReadOnly))
+		useAble = false;
+		hdict = NULL;
+
+		QString fileName = LanguageManager::instance()->getHyphFilename( doc->Language, false );
+		if (!fileName.isEmpty())
 		{
-			QTextStream st(&f);
-			QString line;
-			line = st.readLine();
-			codec = QTextCodec::codecForName(line.toUtf8());
-			f.close();
+			pfad += fileName;
+			QFile f(pfad);
+			if (f.open(QIODevice::ReadOnly))
+			{
+				QTextStream st(&f);
+				QString line;
+				line = st.readLine();
+				codec = QTextCodec::codecForName(line.toUtf8());
+				f.close();
+				useAble = true;
+			}
 		}
-		else
-		{
-			useAble = false;
-			hdict = NULL;
+
+
+		if (!useAble)
 			return;
-		}
+
 		QByteArray fn = pfad.toLocal8Bit();
 		filename = fn.data();
 		hdict = hnj_hyphen_load(filename);
diff -ru scribus-1.3.5.rc2.orig/scribus/hysettings.cpp scribus-1.3.5.rc2/scribus/hysettings.cpp
--- scribus-1.3.5.rc2.orig/scribus/hysettings.cpp	2009-06-15 10:15:20.000000000 +0100
+++ scribus-1.3.5.rc2/scribus/hysettings.cpp	2009-06-15 12:53:18.000000000 +0100
@@ -27,7 +27,17 @@
 	language->setInsertPolicy(QComboBox::InsertAlphabetically);
 	foreach(QString hlang, lmg->hyphLangs())
 	{
-		language->addItem( lmg->getLangFromAbbrev(hlang), lmg->getLangFromAbbrev(hlang,false) );
+		QString uiname = lmg->getLangFromAbbrev(hlang);
+		QString lang = lmg->getLangFromAbbrev(hlang,false);
+
+		//Fallback to use lang code as name if otherwise unknown
+		if (uiname.isEmpty())
+			uiname = hlang;
+
+		if (lang.isEmpty())
+			lang = hlang;
+
+		language->addItem( uiname, lang );
 	}
 	
 	buttonExceptAdd->setIcon(QIcon(loadIcon("16/list-add.png")));
diff -ru scribus-1.3.5.rc2.orig/scribus/langmgr.cpp scribus-1.3.5.rc2/scribus/langmgr.cpp
--- scribus-1.3.5.rc2.orig/scribus/langmgr.cpp	2009-06-15 10:15:15.000000000 +0100
+++ scribus-1.3.5.rc2/scribus/langmgr.cpp	2009-06-15 12:48:39.000000000 +0100
@@ -315,7 +314,31 @@
 {
 	if(langIsAbbreviated)
 		return hyphLangList.value(lang);
-	return hyphLangList.value(getAbbrevFromLang(lang, false));
+
+	QString abbrev = getAbbrevFromLang(lang, false);
+
+	//If we don't have the language in our hard-coded table, 
+	//then try it directly
+	if (abbrev.isEmpty())
+		abbrev = lang;
+	QString ret = hyphLangList.value(abbrev);
+	if (ret.isEmpty())
+	{
+		//As a hack, if the language was munged into an abbreviated
+		//generic version through our table, e.g. German to "de"
+		//then hope for the best and pick the first of e.g.
+		//"de_AT", "de_DE" if they exist
+        	foreach(QString hlang, hyphLangs())
+		{
+			QStringList items = hlang.split("_");
+			if (!items.empty() && items[0] == abbrev)
+			{
+				ret = hyphLangList.value(hlang);
+				break;
+			}
+		}
+	}
+	return ret;
 }
 
 const QStringList LanguageManager::hyphLangs()

sharkcz

2010-10-05 16:47

reporter  

scribus-1.3.8-arbitrary-hyphenators.patch (5,354 bytes)   
From 0ccfe17b41250d1c5f21f46309b60bab31ab0d7e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Dan=20Hor=C3=A1k?= <dan@danny.cz>
Date: Tue, 5 Oct 2010 16:45:49 +0200
Subject: [PATCH] support arbitrary seperately packaged hyphenators

---
 Scribus/scribus/hyphenator.cpp |   62 +++++++++++++++++++++++----------------
 Scribus/scribus/hysettings.cpp |   12 +++++++-
 Scribus/scribus/langmgr.cpp    |   26 ++++++++++++++++-
 Scribus/scribus/scribus.cpp    |    2 +-
 4 files changed, 73 insertions(+), 29 deletions(-)

diff --git a/Scribus/scribus/hyphenator.cpp b/Scribus/scribus/hyphenator.cpp
index 3e0d3d8..38cf5d2 100644
--- a/Scribus/scribus/hyphenator.cpp
+++ b/Scribus/scribus/hyphenator.cpp
@@ -61,23 +61,26 @@ Hyphenator::Hyphenator(QWidget* parent, ScribusDoc *dok) : QObject( parent ),
 		Language = PrefsManager::instance()->appPrefs.Language;
 		doc->Language = Language;
 	}
+
 // 	pfad += ScCore->primaryMainWindow()->Sprachen[Language];
-	pfad += lmgr->getHyphFilename( doc->Language, false );
-	QFile f(pfad);
-	if (f.open(QIODevice::ReadOnly))
+	QString fileName = lmgr->getHyphFilename( doc->Language, false );
+ 	if (!fileName.isEmpty())
 	{
-		QTextStream st(&f);
-    	QString line;
-    	line = st.readLine();
-		codec = QTextCodec::codecForName(line.toUtf8());
-		f.close();
+		pfad += fileName;
+		QFile f(pfad);
+		if (f.open(QIODevice::ReadOnly))
+		{
+			QTextStream st(&f);
+			QString line;
+			line = st.readLine();
+			f.close();
+			useAble = true;
+		}
 	}
-	else
-	{
-		useAble = false;
-		hdict = NULL;
+
+	if (!useAble)
 		return;
-	}
+
 	QByteArray fn = pfad.toLocal8Bit();
 	const char * filename = fn.data();
 	hdict = hnj_hyphen_load(filename);
@@ -111,22 +114,29 @@ void Hyphenator::NewDict(const QString& name)
 		if (hdict != NULL)
 			hnj_hyphen_free(hdict);
 
-		pfad +=  LanguageManager::instance()->getHyphFilename(Language, false) ;
-		QFile f(pfad);
-		if (f.open(QIODevice::ReadOnly))
+		useAble = false;
+		hdict = NULL;
+
+		QString fileName = LanguageManager::instance()->getHyphFilename( doc->Language, false );
+		if (!fileName.isEmpty())
 		{
-			QTextStream st(&f);
-			QString line;
-			line = st.readLine();
-			codec = QTextCodec::codecForName(line.toUtf8());
-			f.close();
+			pfad += fileName;
+			QFile f(pfad);
+			if (f.open(QIODevice::ReadOnly))
+			{
+				QTextStream st(&f);
+				QString line;
+				line = st.readLine();
+				codec = QTextCodec::codecForName(line.toUtf8());
+				f.close();
+				useAble = true;
+			}
 		}
-		else
-		{
-			useAble = false;
-			hdict = NULL;
+
+
+		if (!useAble)
 			return;
-		}
+
 		QByteArray fn = pfad.toLocal8Bit();
 		filename = fn.data();
 		hdict = hnj_hyphen_load(filename);
diff --git a/Scribus/scribus/hysettings.cpp b/Scribus/scribus/hysettings.cpp
index 27dc7b0..ebd0407 100644
--- a/Scribus/scribus/hysettings.cpp
+++ b/Scribus/scribus/hysettings.cpp
@@ -27,7 +27,17 @@ HySettings::HySettings( QWidget* parent/*, QMap<QString,QString>* langs*/ ) : QW
 	language->setInsertPolicy(QComboBox::InsertAlphabetically);
 	foreach(QString hlang, lmg->hyphLangs())
 	{
-		language->addItem( lmg->getLangFromAbbrev(hlang), lmg->getLangFromAbbrev(hlang,false) );
+		QString uiname = lmg->getLangFromAbbrev(hlang);
+		QString lang = lmg->getLangFromAbbrev(hlang,false);
+
+		//Fallback to use lang code as name if otherwise unknown
+		if (uiname.isEmpty())
+			uiname = hlang;
+
+		if (lang.isEmpty())
+			lang = hlang;
+
+		language->addItem( uiname, lang );
 	}
 	
 	buttonExceptAdd->setIcon(QIcon(loadIcon("16/list-add.png")));
diff --git a/Scribus/scribus/langmgr.cpp b/Scribus/scribus/langmgr.cpp
index 6167c50..cd510b1 100644
--- a/Scribus/scribus/langmgr.cpp
+++ b/Scribus/scribus/langmgr.cpp
@@ -315,7 +315,31 @@ const QString LanguageManager::getHyphFilename(const QString & lang, bool langIs
 {
 	if(langIsAbbreviated)
 		return hyphLangList.value(lang);
-	return hyphLangList.value(getAbbrevFromLang(lang, false));
+
+	QString abbrev = getAbbrevFromLang(lang, false);
+
+	//If we don't have the language in our hard-coded table, 
+	//then try it directly
+	if (abbrev.isEmpty())
+		abbrev = lang;
+	QString ret = hyphLangList.value(abbrev);
+	if (ret.isEmpty())
+	{
+		//As a hack, if the language was munged into an abbreviated
+		//generic version through our table, e.g. German to "de"
+		//then hope for the best and pick the first of e.g.
+		//"de_AT", "de_DE" if they exist
+        	foreach(QString hlang, hyphLangs())
+		{
+			QStringList items = hlang.split("_");
+			if (!items.empty() && items[0] == abbrev)
+			{
+				ret = hyphLangList.value(hlang);
+				break;
+			}
+		}
+	}
+	return ret;
 }
 
 const QStringList LanguageManager::hyphLangs()
diff --git a/Scribus/scribus/scribus.cpp b/Scribus/scribus/scribus.cpp
index 6b8fb9c..5f3aef2 100644
--- a/Scribus/scribus/scribus.cpp
+++ b/Scribus/scribus/scribus.cpp
@@ -8899,7 +8899,7 @@ void ScribusMainWindow::initHyphenator()
 	//Grab the language abbreviation from it, get the full language text
 	//Insert the name as key and a new string list into the map
 	QString hyphDirName = QDir::convertSeparators(ScPaths::instance().dictDir());
-	QDir hyphDir(hyphDirName, "*.dic", QDir::Name, QDir::Files | QDir::NoSymLinks);
+	QDir hyphDir(hyphDirName, "*.dic", QDir::Name, QDir::Files);
 	if ((hyphDir.exists()) && (hyphDir.count() != 0))
 	{
 // 		LanguageManager langmgr;
-- 
1.7.2.3

cbradney

2010-10-07 20:52

administrator   ~0024621

Currently testing building with this patch. I have removed the incorrect Galician references. Once working, this needs fwd porting to 1.5.0svn

cbradney

2010-10-07 20:58

administrator   ~0024622

Does the hyphen directory get created by an Mozilla install or some independent package? I am wondering for other platforms like OSX or Windows..

sharkcz

2010-10-08 15:44

reporter   ~0024627

The hyphenation dict, spelling dict and the applications/libraries are installed (at least in Fedora/RHEL/CentOS) as a part of "XY Language support" during the OS installation. Additional languages can be installed later using the "Add/Remove software" tool or directly via package management from the hyphen-* or aspell-*/hunspell-* packages. I'd expect other distributions use similar solution.

In my opinion Scribus can use a check in ScPaths::dictDir() - if m_shareDir + "dicts/" exists then use this as dictDir, otherwise use the system dicst from "/usr/share/hyphen" (distribution package will remove scribus provided dicts)

cbradney

2010-10-09 12:25

administrator   ~0024633

Last edited: 2010-10-09 12:29

I am still confused as to how we can provide for these hyphenation dictionaries on other platforms.. dicts will always exist as we always install to it.

It is not at all clear how to get hunspell dicts for OSX or Windows unless we just download them from somewhere like http://wiki.services.openoffice.org/wiki/Dictionaries. With Macports version of hunspell,

hunspell-dict-af_ZA @2006-01-17 (textproc)
hunspell-dict-ca_ES @2002-10-15 (textproc)
hunspell-dict-cs_CZ @2003-01-01 (textproc)
hunspell-dict-cy_GB @2004-04-25 (textproc)
hunspell-dict-de_DE @2006-02-07 (textproc)
hunspell-dict-el_GR @2004-12-20 (textproc)
hunspell-dict-en_CA @2002-03-15 (textproc)
hunspell-dict-en_NZ @2002-05-18 (textproc)
hunspell-dict-en_US @2006-02-07 (textproc)
hunspell-dict-en_ZA @2006-01-20 (textproc)
hunspell-dict-eo_EO @2005-07-10 (textproc)
hunspell-dict-es_ES @2005-05-10 (textproc)
hunspell-dict-es_MX @2005-05-05 (textproc)
hunspell-dict-fo_FO @2005-03-07 (textproc)
hunspell-dict-fy_NL @2007-11-28 (textproc)
hunspell-dict-ga_IE @2007-10-29 (textproc)
hunspell-dict-gd_GB @2005-01-08 (textproc)
hunspell-dict-gsc_FR @2007-08-16 (textproc)
hunspell-dict-gu_IN @2006-10-15 (textproc)
hunspell-dict-he_IL @2005-01-12 (textproc)
hunspell-dict-hi_IN @2007-02-19 (textproc)
hunspell-dict-hr_HR @2006-06-07 (textproc)
hunspell-dict-id_ID @2004-08-12 (textproc)
hunspell-dict-ku_TR @2005-01-21 (textproc)
hunspell-dict-lt_LT @2003-12-31 (textproc)
hunspell-dict-mg_MG @2005-01-09 (textproc)
hunspell-dict-mk_MK @2005-11-26 (textproc)
hunspell-dict-ms_MY @2005-01-17 (textproc)
hunspell-dict-nb_NO @2006-05-08 (textproc)
hunspell-dict-nl_NL @2007-06-07 (textproc)
hunspell-dict-nn_NO @2006-05-08 (textproc)
hunspell-dict-nr_ZA @2006-01-20 (textproc)
hunspell-dict-ns_ZA @2006-01-20 (textproc)
hunspell-dict-ny_MW @2005-01-08 (textproc)
hunspell-dict-oc_FR @2006-09-22 (textproc)
hunspell-dict-rw_RW @2005-01-08 (textproc)
hunspell-dict-sl_SI @2007-01-27 (textproc)
hunspell-dict-ss_ZA @2006-07-05 (textproc)
hunspell-dict-st_ZA @2006-01-20 (textproc)
hunspell-dict-sw_KE @2004-05-16 (textproc)
hunspell-dict-tet_ID @2005-01-08 (textproc)
hunspell-dict-th_TH @2006-12-12 (textproc)
hunspell-dict-tl_PH @2005-01-08 (textproc)
hunspell-dict-tn_ZA @2004-05-16 (textproc)
hunspell-dict-ts_ZA @2006-01-23 (textproc)
hunspell-dict-uk_UA @2009-01-25 (textproc)
hunspell-dict-ve_ZA @2006-07-06 (textproc)
hunspell-dict-xh_ZA @2006-01-23 (textproc)
hunspell-dict-zu_ZA @2006-01-20 (textproc)

Issue History

Date Modified Username Field Change
2009-06-15 13:20 caolan New Issue
2009-06-15 13:20 caolan File Added: scribus-1.3.5.rc2.arbitraryhyphenators.patch
2009-07-30 23:06 plinnell Status new => assigned
2009-07-30 23:06 plinnell Assigned To => cbradney
2010-10-05 16:47 sharkcz File Added: scribus-1.3.8-arbitrary-hyphenators.patch
2010-10-07 20:52 cbradney Note Added: 0024621
2010-10-07 20:58 cbradney Note Added: 0024622
2010-10-08 15:45 sharkcz Note Added: 0024627
2010-10-09 12:25 cbradney Note Added: 0024633
2010-10-09 12:29 cbradney Note Edited: 0024633
2014-08-31 03:09 Kunda Summary Support for arbitrary seperately packaged hyphenators => [PATCH[ Support for arbitrary seperately packaged hyphenators
2014-08-31 03:10 Kunda Summary [PATCH[ Support for arbitrary seperately packaged hyphenators => [PATCH] Support for arbitrary seperately packaged hyphenators
2014-09-03 17:33 cbradney Status assigned => closed
2014-09-03 17:33 cbradney Resolution open => won't fix
2014-10-24 22:57 Kunda Patch => Yes