View Issue Details

IDProjectCategoryView StatusLast Update
0016447ScribusPDFpublic2021-01-27 11:03
Reportertoyotadesigner Assigned Tojghali  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformmacOSOSCatalinaOS Version10.15.7 (19H114)
Product Version1.5.7.svn 
Target VersionFixed in Version1.5.7.svn 
Summary0016447: Ligatures cannot be extracted from PDF as text
DescriptionWe are currently publishing a brochure. While working on the project, we export single pages to manually print and produce mock-ups.
We import the PDF page into PhotoLine to place it at the top of a DIN A4 page, because the brochure features a smaller format and macOS preview places it in the center of a page.
After the import into PhotoLine, the ligature »ff« is invisable and does not print (see samples below)
I contacted the developer of PhotoLine, and they informed me, that the ligature is coded incorrectly.
So I opened the PDF with Affinity Designer, and instead of the ligature »ff« there is a weird character displayed and printed.

Additional InformationScreenshots:
1. Ligatures disabled
2. Ligatures enabled, as displayed in PhotoLine
3. Ligatures enabled, as displayed in Affinity Designer
Tagswrong ligatures
PatchNo

Activities

toyotadesigner

2021-01-25 07:40

reporter  

with_ligatures_PL.png (32,694 bytes)
with_ligatures_PL.png (32,694 bytes)
without_ligatures_PL.png (33,083 bytes)
without_ligatures_PL.png (33,083 bytes)

jghali

2021-01-25 10:59

administrator   ~0048754

Please provide sample document and font.

toyotadesigner

2021-01-25 11:13

reporter   ~0048755

I'll attach

- *.sla file
- PDF
- font file

Raleway-v4020-Regular.otf (126,700 bytes)
vorwort_grid_2021-01-22_ligatures.sla (23,344 bytes)
vorwort_grid_2021-01-23_ligatures.pdf (1,407,811 bytes)

jghali

2021-01-25 13:51

administrator   ~0048756

Last edited: 2021-01-25 14:26

View 2 revisions

>> I contacted the developer of PhotoLine, and they informed me, that the ligature is coded incorrectly.

After analysis, this is highly unlikely. The font embedded inside PDF uses an Identity encoding, which means we output text by using glyph indexes directly. A PDF viewer in such case has basically no translation work to do: it retrieves glyph indexes, get directly glyphs from font using glyph table and displays them. There is no translation from character codes to glyph indexes to do. Without having the Raleway font installed, Acrobat also displays and print the ligature without issue. Adobe Indesign can also import your sample Scribus PDF and export it again without any issue with ligatures.

My guess is that PhotoLine and Affinity Designer are somehow trying to get the original text from the glyphs indexes stored in the PDF while importing. That would be a fundamental error as PDF format does not provide a guarantee of reliability when trying to do that. Importing PDF files in other softwares (except specialized prepress workflows) has always been error prone. Even Adobe Illustrator has some big bugs with PDF import. So if you're trying to import PDF files in software such as PhotoLine or Affinity Design, whose primary purpose is not the processing of PDF files, I would strongly suggest that you fully outline fonts instead of embedding or subsetting them.

toyotadesigner

2021-01-25 14:51

reporter   ~0048757

Dear jghali, thank you very much for looking into this issue and describing the problem. If you don't mind, I will send your information to the developers of PhotoLine and Affinity.

jghali

2021-01-25 22:47

administrator   ~0048758

>> If you don't mind, I will send your information to the developers of PhotoLine and Affinity.

I don't mind :-)

toyotadesigner

2021-01-26 16:23

reporter   ~0048761

OK, I've sent your information to the developers of PhotoLine. They found the bug in Scribus and replied as follows:

After analysis, this is highly unlikely. The font embedded inside PDF uses an Identity encoding, which means we output text by using glyph indexes directly. A PDF viewer in such case has basically no translation work to do: it retrieves glyph indexes, get directly glyphs from font using glyph table and displays them.
-----
Reply:
Soweit richtig. / This is correct.
-----
There is no translation from character codes to glyph indexes to do.
-----
Reply:
Auch das ist richtig. Aber Um den Text zu dekodieren, muss man von einem Glyph Index zu einem oder mehreren Character Codes kommen können. / This is also correct. But to decode the text, one must be able to get from a glyph index to one or more character codes.
-----
Without having the Raleway font installed, Acrobat also displays and print the ligature without issue. Adobe Indesign can also import your sample Scribus PDF and export it again without any issue with ligatures.
My guess is that PhotoLine and Affinity Designer are somehow trying to get the original text from the glyphs indexes stored in the PDF while importing.
-----
Reply:
Auch richtig. / Correct as well.
-----
That would be a fundamental error as PDF format does not provide a guarantee of reliability when trying to do that.
-----
Reply:
Das ist falsch. Es gibt für diesen Fall zum einen vordefinierte Encodings oder die ToUnicode-Tabelle (die Scribus verwendet, aber dort einen falschen Eintrag für die Ligatur enthält). Die PDF Spezifikation ISO 32000-1 beschreibt das in Kapitel "9.10 Extraction of Text Content".

PDF/A basiert darauf, dass PDF-Dateien dekodierbaren Text enthalten. /

This is wrong. There are predefined encodings for this case, or the ToUnicode table (which Scribus uses, but contains an incorrect entry for the ligature). The PDF specification ISO 32000-1 (http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf) describes this in chapter "9.10 Extraction of Text Content".

PDF/A is based on the fact that PDF files contain decodable text.

Maybe this is of help for you…

jghali

2021-01-26 16:58

administrator   ~0048763

Last edited: 2021-01-26 17:29

View 4 revisions

>>> That would be a fundamental error as PDF format does not provide a guarantee of reliability when trying to do that.
-----
Reply:
Das ist falsch. Es gibt für diesen Fall zum einen vordefinierte Encodings oder die ToUnicode-Tabelle (die Scribus verwendet, aber dort einen falschen Eintrag für die Ligatur enthält). Die PDF Spezifikation ISO 32000-1 beschreibt das in Kapitel "9.10 Extraction of Text Content".

This is incorrect. The ToUnicode table is completely optional in the PDF standard. See section "9.6.2 Type 1 Fonts", "9.6.5 Type 3 Fonts". "9.7.6 Type 0 Font Dictionaries". So text extractability is not guaranteed at all in the PDF format. Trying to use the ToUnicode table to import PDF is not necessary to render PDF text correctly and is incorrect, especially when PDF font use an Identity encoding.

The "9.10 Extraction of Text Content" section of PDF standard mentions itself that there are situations where original text may not be extracted.

>> This is wrong. There are predefined encodings for this case, or the ToUnicode table (which Scribus uses, but contains an incorrect entry for the ligature). The PDF specification ISO 32000-1 (http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf) describes this in chapter "9.10 Extraction of Text Content".

The ToUnicode table is optional and must not be used to render PDF text. So any reference to that 9.10 chapter does not apply. Try for eg to import your PDF by opening it directly in Scribus, we import the ligature correctly. Adobe InDesign import it correctly too. Beginning of 9.10.1 section of PDF standard also clearly says:
"The preceding sub-clauses describe all the facilities for showing text and causing glyphs to be painted on the page."
Which clearly means that the ToUnicode table is not needed to get PDF text right on screen.

>> PDF/A is based on the fact that PDF files contain decodable text.

Scribus does not support PDF/A standard and does not export to that format. PDF/X and PDF/A are different. So that remark about text extractability does not apply to our case.

jghali

2021-01-26 18:20

administrator   ~0048768

Last edited: 2021-01-26 18:47

View 3 revisions

Another case where their PDF import method would probably fail: OpenType alternate glyphs. In such case several glyphs of a font can map to same unicode value. Using the ToUnicode table to get the original text then trying from the text to get same glyph as PDF has all chances to fail as the PDF format has no knowledge about the OpenType layout options necessary to get back to the same glyph as used in the PDF.

In the same way, that argument about the necessity to get a unicode value for ligatures is bogus as most ligatures have no assigned value in the unicode standard. The Raleway font rightfully does not provide any character code for the ff ligature so the best we could do would be to provide unicode values for two 'f' characters in the ToUnicode table. Without any knowledge of the OpenType options used while doing layout in Scribus, I doubt they are able to get back to the proper glyph reliably from the original text.

toyotadesigner

2021-01-26 20:38

reporter   ~0048771

Thank you very much for the information. I'll turn off the ligature option for future projects to remain on the safe side.

jghali

2021-01-26 22:04

administrator   ~0048772

>> I'll turn off the ligature option for future projects to remain on the safe side.

As mentioned in a previous note, you could switch the font embedding mode to "Outline All Fonts". In that case, you'll get the ligatures in your PDF and the risk that the ligature disappears when importing your PDF in another software will be reduced to close to 0.

jghali

2021-01-26 23:08

administrator   ~0048773

I did some big changes to improve text extractability. When copying text from PDF, now ligatures will be extracted using their character decomposition, ie "ff" ligatures will be represented with two "f" characters when pasted in a text editor.

Issue History

Date Modified Username Field Change
2021-01-25 07:40 toyotadesigner New Issue
2021-01-25 07:40 toyotadesigner Tag Attached: wrong ligatures
2021-01-25 07:40 toyotadesigner File Added: with_ligatures_PL.png
2021-01-25 07:40 toyotadesigner File Added: without_ligatures_PL.png
2021-01-25 07:40 toyotadesigner File Added: wrong_ligature_in_affinity.png
2021-01-25 10:59 jghali Note Added: 0048754
2021-01-25 11:13 toyotadesigner File Added: Raleway-v4020-Regular.otf
2021-01-25 11:13 toyotadesigner File Added: vorwort_grid_2021-01-22_ligatures.sla
2021-01-25 11:13 toyotadesigner File Added: vorwort_grid_2021-01-23_ligatures.pdf
2021-01-25 11:13 toyotadesigner Note Added: 0048755
2021-01-25 13:51 jghali Note Added: 0048756
2021-01-25 14:26 jghali Note Edited: 0048756 View Revisions
2021-01-25 14:51 toyotadesigner Note Added: 0048757
2021-01-25 22:47 jghali Note Added: 0048758
2021-01-26 16:23 toyotadesigner Note Added: 0048761
2021-01-26 16:58 jghali Note Added: 0048763
2021-01-26 17:07 jghali Note Edited: 0048763 View Revisions
2021-01-26 17:08 jghali Note Edited: 0048763 View Revisions
2021-01-26 17:29 jghali Note Edited: 0048763 View Revisions
2021-01-26 18:20 jghali Note Added: 0048768
2021-01-26 18:25 jghali Note Edited: 0048768 View Revisions
2021-01-26 18:47 jghali Note Edited: 0048768 View Revisions
2021-01-26 20:38 toyotadesigner Note Added: 0048771
2021-01-26 22:04 jghali Note Added: 0048772
2021-01-26 23:03 jghali Summary ligatures coded incorrectly => Ligatures cannot be extracted from PDF as text
2021-01-26 23:03 jghali Description Updated View Revisions
2021-01-26 23:03 jghali Additional Information Updated View Revisions
2021-01-26 23:08 jghali Assigned To => jghali
2021-01-26 23:08 jghali Status new => resolved
2021-01-26 23:08 jghali Resolution open => fixed
2021-01-26 23:08 jghali Fixed in Version => 1.5.7.svn
2021-01-26 23:08 jghali Note Added: 0048773
2021-01-27 01:23 jghali Severity major => minor
2021-01-27 11:03 jghali Category Import / Export => PDF