View Issue Details

IDProjectCategoryView StatusLast Update
0016142ScribusImport / Exportpublic2020-10-02 21:27
Reporteroliverthered Assigned To 
Status newResolutionopen 
PlatformallOSOS Version
Product Version1.5.6.svn 
Target VersionFixed in Version 
Summary0016142: review and merge pull request 0000135 to begin supporting text import from PDF files
DescriptionI have created the following pull request from my fork of the Scribus project:

Ale and I have been working on getting it ready over the last couple of weeks and I'm fairly confident it's good for reviewing and integrating with the main branch.

It adds basic PDF text import functionality, just the text and basic layout functionality, no styles or fonts, right or centere3d justification, rotation, multipage etc... but it does provide enough of a base that most of the extra required functionality can be worked on as isolated features with low interdependencies.

I hope the code is to your standard and liking, and we can get it merged quickly, so I can get on writing the extra required features.
Steps To ReproduceImport a vector file in pdf format.
Note that the text is vectors and you are unable to edit it in and reasonable fashion.
Additional InformationA final review and sign off by ale may be required.
Tagsimport, PDF, pdf import, vectorised fonts


has duplicate 0016129 closedjghali basic textframe and text rendering 
has duplicate 0016128 closedjghali implement some more on the basic pdf text import, framework, abstract addchar and ui for selecting import method 
has duplicate 0016126 closedjghali refactor getChar so that it can be used for importing text as well as drawing vector characters 
has duplicate 0016125 closedjghali Implement a framework for importing text from pdf documents 
has duplicate 0016127 closedjghali override updateTextPos in SlaOutputDev 
has duplicate 0016103 closedjghali Update PDF import with features matching Inkscapes PDF import 



2020-06-21 13:06

administrator   ~0047710

I had a quick look. The renaming of updateFont() method to updateFontForVector() looks wrong. The updateFont() method is indeed a virtual method of poppler OutputDev class which is called by poppler itself and needs to be reimplemented at least for the text as vectors case.


2020-06-21 13:13

reporter   ~0047711

ok, sounds like a plan. most of the functionality of SlaOutputDev needs to remain the same so I'll create a new outputdev class and inherit from SlaOutputDev overriding the methods needed for text import and just choose the correct outputdev class in bool PdfPlug::convert(const QString& fn)


2020-06-21 13:24

administrator   ~0047712

Hmm, I think this is a bit overkill. You will need that updateFont() method anyway to get proper font information in the end.


2020-06-21 13:53

reporter   ~0047713

oh, I've done it. I'll tidy the code up and see what you think.


2020-06-21 13:55

reporter   ~0047714

laOutputDev::updateFont loads the fonts from the pdf file I think, which I don't want I need to use scribus accessible fonts and will probably load all the fonts from the document and allow substitutions to be set in the pdf import UI. well, that's my current thoughts, font overrides need to happen some place and should be user configurable.


2020-06-21 14:11

reporter   ~0047715

I think implementing a new OutputDev based on SlaOutputDev gives the tidiest implementation and doesn't really have any overheads.


2020-06-21 14:11

reporter   ~0047716

I've pushed the changes now.


2020-06-21 15:18

reporter   ~0047717

the other consideration is that the implementation would be nice if it were easy to use in other projects without too much modification. so keeping it all encapsulated in it's own outputdev as opposed to being incorporated in SlaOutputDev achieves that goal too.


2020-06-21 15:35

administrator   ~0047718

TextOutputDev is already a poppler class name, you should better avoid it.


2020-06-21 15:41

reporter   ~0047719

I've had a thought and there are a few more things from SlaOiutputDev that need to be implemented differently for importing text as opposed to vectors so cresting a new class makes sense. I'll change the name.


2020-06-21 15:44

reporter   ~0047720

ok, the name from TextOutputDev to PdfTextOutputDev.
the PdfTextOutputDev naming matches the naming of PdfTextRecognition which is why I chose it.


2020-06-21 21:18

reporter   ~0047721

I've gone through and renames all the classes and member variables so they all start with 'PdfText' for consistency


2020-06-26 15:50


pdf-text-import-phase1.rar (12,937 bytes)


2020-06-26 15:51

reporter   ~0047736

this patch should be good to go. there's a couple of bugs i fixed upstream but they should be in the next patch, the bugs don't stop functionanlity etc..


2020-06-26 16:11

reporter   ~0047737

this patch should be better. it makes minimal changes to slaoutputdev and fixes a bug

pdf-text-import-phase1-2.rar (12,564 bytes)


2020-06-26 21:44

administrator   ~0047738

Unfortunately I cannot apply this patch:
patch: **** malformed patch at line 1015: @@ -3359,8 +3341,9 @@ void SlaOutputDev::drawChar(GfxState *state, double x, double y, double dx, doub


2020-06-26 22:05

administrator   ~0047739

I finished the merge manually. I started doing some tests and found a bug : in most cases I do not see the dialog which would allow me to choose how text should be imported.


2020-06-26 22:14

administrator   ~0047741

There are lots of missing braces in PdfTextRegion::linearTest() too


2020-06-26 22:35

administrator   ~0047742

Hmmm, in most of my tests, when importing text as text the imported text is often not even imported at an approximately correct position.


2020-06-26 22:42

administrator   ~0047743

Try this file for example;
When importing text as text, import fails pretty miserably :(


2020-07-02 06:48

reporter   ~0047751

my current working version that supports font and styles isn't too far off. I've not added support for displacement which is different from just the text position I'm currently using.

2020-07-02 (3).png (1,031,872 bytes)
2020-07-02 (3).png (1,031,872 bytes)


2020-07-02 06:58

reporter   ~0047752

most of the pdf files I have for testing at my end were all probably generated by latex. it would be good to get a broader selection of pdf files from different sources.


2020-07-02 07:00

reporter   ~0047753

What are the requirement for getting the code merged with scribus.

I can add some more braces. ale said that there should be no braces if the if or else following is only one line.


2020-07-02 07:09

reporter   ~0047754

and for some of the documents a second parse over the layout is required because the text is separated by lots of begin/end text regions that should be merged together as a continuous block of text.


2020-07-02 07:17

reporter   ~0047755

my aim with the first patch certainly isn't perfection but enough of a base that features can be added to it fairly cleanly.


2020-07-02 08:37

reporter   ~0047756

here's a screenshot of my work in progress using the pdf file i've been using for the bulk of testing. After I've fixed the colours so they are presented correctly there should only be smaller features and bug fixes to be implemented.


2020-07-02 10:44

reporter   ~0047757

here's what the index page of the attached pdf look like in my current working version:


2020-07-02 10:50

reporter   ~0047758

oops, that's imported as vectors not text.


2020-07-02 10:59

reporter   ~0047759

this is what it actually looks like :
the baseline is correct it's just creating textframes that are too tall which makes it look more of a mess than it actually is.


2020-07-02 11:15

administrator   ~0047760

>> I can add some more braces. ale said that there should be no braces if the if or else following is only one line.

In PdfTextRegion::linearTest() you have code like this:

if (collinear(point.y(), lastXY.y()))
        if (collinear(point.x(), lastXY.x()))
            return LineType::FIRSTPOINT;
        else if (xInLimits)
            return LineType::SAMELINE;
            qDebug() << ...;

It's clear that the block related to the if (collinear(point.y(), lastXY.y())) condition uses more than one line, so code should be inside braces:

if (collinear(point.y(), lastXY.y()))
        if (collinear(point.x(), lastXY.x()))
            return LineType::FIRSTPOINT;
        else if (xInLimits)
            return LineType::SAMELINE;
            qDebug() << ...;

You have similar blocks in same function which should be enclosed in braces.


2020-07-02 11:38

reporter   ~0047762

here's what it looks like after a small tweek:
to get it looking correct i'll have to lookup the linespacing of the first line from the font instead of making an assumption based on a multiple of dx.


2020-07-02 12:03

reporter   ~0047763

setting the linespacing based on the font height instead of a multiple of dx seems to have done the trick. but that's in the fonts and styles update not in the patch for this issue.

2020-07-02 (13).png (551,962 bytes)
2020-07-02 (13).png (551,962 bytes)
2020-07-02 (12).png (272,812 bytes)
2020-07-02 (12).png (272,812 bytes)


2020-07-02 14:54

reporter   ~0047764

I can backport some of the bug fixes that i did for fonts and styles into the initial patch if you want to have the first patch in better working order but the main fix for the attached pdf was replacing some of my guestimates with more intelligent ones based upon the font in use.


2020-07-02 15:14

reporter   ~0047765

ok, i'll do the braces in a few minutes. is there anything else?


2020-07-02 15:33

reporter   ~0047766

find patch attached. I've compiled and run this one so it should be ok this time. I checked and changes to support the pdf document attached to this thread require font support to get anything like a sensible import.

pdftext-phase1.rar (13,453 bytes)


2020-07-02 16:51

reporter   ~0047767

I've had a look at the attached pdf file calls beginTextObject and endTextObject for pretty much every letter which is why there not getting merged. my plan for this was to import all the textframes for a page and then parse over them merging adjacent textframes or textframes that make up new lines in multiline text.


2020-07-02 17:31

reporter   ~0047768

if I ignore begin/end text object and the cursor being reset to the origin I get the following result:

This is the result I'd expect after parsing over all the textframes and merging appropriately.

2020-07-02 (15).png (327,659 bytes)
2020-07-02 (15).png (327,659 bytes)


2020-07-02 21:22

reporter   ~0047770

I'm currently implementing the code to do the merging. that will mean the attached pdf file should import properly.


2020-07-03 05:24

reporter   ~0047771

I got the merge code working successfully. There are still some flow control issues e.g. no line breaks. But they're not implemented anywhere yet. See attached screenshot.

2020-07-03.png (401,107 bytes)
2020-07-03.png (401,107 bytes)


2020-07-03 16:05

reporter   ~0047774

I'm going to add some bespoke code to manage tables as they are quite a common feature and it would be nice to have them laid out properly.


2020-07-03 17:26

administrator   ~0047778

I see one last major issue with the last patch you uploaded: when you open (with file > open) a PDF containing multiple pages, after first page text frames are not created at correct position. In fact they seem to be placed before first page.


2020-07-03 17:34

reporter   ~0047780

yeah, multiple pages aren't supported at the moment. it's on my list as the next thing to support.


2020-07-03 17:36

reporter   ~0047781

my comment in the description for this bug was "It adds basic PDF text import functionality, just the text and basic layout functionality, no styles or fonts, right or centere3d justification, rotation, multipage etc... but it does provide enough of a base that most of the extra required functionality can be worked on as isolated features with low interdependencies."


2020-07-03 17:56

reporter   ~0047782

I've a few more patches all but ready too. one for supporting fonts and styles, one to support merging of erratically placed content into textframes, and one that adds spaces to the end of the lines and line breaks so the text flows properly.


2020-07-03 18:42

reporter   ~0047785

I've just been focused on getting the text rendered in textframes correctly and making sure the code that does it is robust and versatile enough to support all the features required.


2020-07-03 20:11

reporter   ~0047786

ok, multipage support is a quick fix. I'll provide an updated patch. PDF files seem to be read only when opened as files as opposed to being imported as vector files.


2020-07-03 20:25

reporter   ~0047787

find the attached path with multipage support. currently, y is offset by linespace when it should only be offset by the fonts ascending value which is slightly smaller.

pdftext-import-phase1.rar (13,469 bytes)


2020-07-06 17:35

reporter   ~0047791

this patch additionally fixes a problem with the grouping/z-order when importing multiple pages.

pdf-text-import-phase-1-with-multipage-support.rar (13,475 bytes)


2020-07-06 17:45

reporter   ~0047792

that patch breaks importing individual pages.


2020-07-06 18:54

reporter   ~0047793

this patch is correct

pdf-text-import-phase-1-with-multipage-support-2.rar (13,504 bytes)


2020-07-08 00:03

reporter   ~0047799

side by side with acrobat


2020-07-08 00:04


2020-07-08.png (1,193,579 bytes)


2020-07-11 01:05

reporter   ~0047811

I'm going to stop working on PDF whilst I think about patent issues. I'm aggregating the text not placing it as PDF documents do so I doubt there will be any issues
 I'm going to take on a-l-e 's idea of placement due to this too and it was probably a good idea.
in theory you can always code around a patent, though it may not be elegant


2020-07-11 01:22

reporter   ~0047812

I also inadvertently breached copyright, but I think it's worth while giving the paper a little promotion as it's a good one.


2020-07-11 16:56

reporter   ~0047817

I'm going to put together a feature list. I'll use my commits to git to put it together and also provide a list of known outstanding features.


2020-07-11 16:59

reporter   ~0047818

i didn't breach copyright in the came under uk fair use laws.


2020-07-11 17:35

reporter   ~0047819

******in this patch******
Merging free flow text into editable text frames
layout of text frames on the page matching the pdf document.
Correct scaling and width.
Support for multiple pages
******todo, including and initially features in other waiting patches******
UI for selecting text or vectors
automated font substitution
manual override font substitution
styles and fonts
font scaling pixels to points
font scaling substitute font sizes to pdf font sizes
subscript text
superscript text
switching between subscript superscript and normal text
use mode average lineheight for the frame
bold and italic
removal of excess spaces from scanned documents
merging of text from scanned documents
flow-control: inserting of spaces at end of lines
flow-control: inserting of new paragraph markers
flow-control: inserting of carriage returns where needed at end of lines
flow-control: inserting of spaces where letter displacement is a significant amount greater than the letter width
getters and setters
use scfaces for fonts
font substitution including the font style
text overflows: extend textframes to match n*the lineheight
text overflows: nudge remaining overflowing textframes by one em width
iterate fonts at startup to preprepare font substitutions

******that's the list of completed features to merge.******

******this is the list of remaining features to add******

determine the kind of layout a text frame/textframes have.
support for grid-form layouts
support for pageheight when importing single pages
support for indented text to the right
support for indented text to the left
support for rotated text
removal of extra spaces added at the end of break characters.
better handling when text flows from one frame to the next with more than a line break inbetween
some font tidyups add some support for bold and italic that is missing
some font tidyups ensure scaling of text to match pdf text is working in all cases
recognize text as left aligned
recognize text as right aligned
recognise text as centred
support for justified text.
Recognize text that has an end of line less than the size of the first word on the next line as being a kind of list.

****** that's the end of the list ******


2020-07-11 17:48

reporter   ~0047820

matching columns and text continuation and adding scribus frame continuation links.


2020-07-11 22:11

reporter   ~0047822

can you wait a few days before patching please. minior legal issues again. but positive ones this time.


2020-07-12 00:02

administrator   ~0047823

Ok, I'll wait then.


2020-07-27 13:20

administrator   ~0047880

>> can you wait a few days before patching please. minor legal issues again.

Where are we now? Could you make the situation clear by explaining it here?


2020-10-02 17:58

reporter   ~0048123

I would be very interested in this feature (many thanks @oliverthered for implementing it). As far as I know, there currently is no other libre / open-source program which supports as much PDF editing as Scribus. The only thing which is missing to make it a fully-featured PDF editor are the text recognition capabilities.
I hope/think patching should be safe - a few days are definitely over and really I can't see any legal issues.
Kind regards and thanks for this useful software!


2020-10-02 21:27

administrator   ~0048124

>> The only thing which is missing to make it a fully-featured PDF editor are the text recognition capabilities.

Well, don't have too much expectation here : in lots of cases, importing text as text will not allow the original PDF visual aspect to be preserved unless you have the fonts embedded in the PDF installed on your system.

Issue History

Date Modified Username Field Change
2020-06-21 12:17 oliverthered New Issue
2020-06-21 12:17 oliverthered Tag Attached: import
2020-06-21 12:17 oliverthered Tag Attached: PDF
2020-06-21 12:17 oliverthered Tag Attached: pdf import
2020-06-21 12:17 oliverthered Tag Attached: vectorised fonts
2020-06-21 12:39 jghali Relationship added has duplicate 0016129
2020-06-21 12:39 jghali Relationship added has duplicate 0016128
2020-06-21 12:39 jghali Relationship added has duplicate 0016126
2020-06-21 12:40 jghali Relationship added has duplicate 0016125
2020-06-21 12:40 jghali Relationship added has duplicate 0016127
2020-06-21 12:56 jghali Relationship added has duplicate 0016103
2020-06-21 13:06 jghali Note Added: 0047710
2020-06-21 13:13 oliverthered Note Added: 0047711
2020-06-21 13:24 jghali Note Added: 0047712
2020-06-21 13:53 oliverthered Note Added: 0047713
2020-06-21 13:55 oliverthered Note Added: 0047714
2020-06-21 14:11 oliverthered Note Added: 0047715
2020-06-21 14:11 oliverthered Note Added: 0047716
2020-06-21 15:18 oliverthered Note Added: 0047717
2020-06-21 15:35 jghali Note Added: 0047718
2020-06-21 15:41 oliverthered Note Added: 0047719
2020-06-21 15:44 oliverthered Note Added: 0047720
2020-06-21 21:18 oliverthered Note Added: 0047721
2020-06-26 15:50 oliverthered File Added: pdf-text-import-phase1.rar
2020-06-26 15:51 oliverthered Note Added: 0047736
2020-06-26 16:11 oliverthered File Added: pdf-text-import-phase1-2.rar
2020-06-26 16:11 oliverthered Note Added: 0047737
2020-06-26 21:44 jghali Note Added: 0047738
2020-06-26 22:01 jghali Priority high => normal
2020-06-26 22:05 jghali Note Added: 0047739
2020-06-26 22:14 jghali Note Added: 0047741
2020-06-26 22:35 jghali Note Added: 0047742
2020-06-26 22:42 jghali Note Added: 0047743
2020-07-02 06:48 oliverthered File Added: 2020-07-02 (3).png
2020-07-02 06:48 oliverthered Note Added: 0047751
2020-07-02 06:58 oliverthered Note Added: 0047752
2020-07-02 07:00 oliverthered Note Added: 0047753
2020-07-02 07:09 oliverthered Note Added: 0047754
2020-07-02 07:17 oliverthered Note Added: 0047755
2020-07-02 08:37 oliverthered Note Added: 0047756
2020-07-02 10:44 oliverthered Note Added: 0047757
2020-07-02 10:50 oliverthered Note Added: 0047758
2020-07-02 10:59 oliverthered Note Added: 0047759
2020-07-02 11:15 jghali Note Added: 0047760
2020-07-02 11:38 oliverthered Note Added: 0047762
2020-07-02 12:03 oliverthered File Added: 2020-07-02 (13).png
2020-07-02 12:03 oliverthered File Added: 2020-07-02 (12).png
2020-07-02 12:03 oliverthered Note Added: 0047763
2020-07-02 14:54 oliverthered Note Added: 0047764
2020-07-02 15:14 oliverthered Note Added: 0047765
2020-07-02 15:33 oliverthered File Added: pdftext-phase1.rar
2020-07-02 15:33 oliverthered Note Added: 0047766
2020-07-02 16:51 oliverthered Note Added: 0047767
2020-07-02 17:31 oliverthered File Added: 2020-07-02 (15).png
2020-07-02 17:31 oliverthered Note Added: 0047768
2020-07-02 21:22 oliverthered Note Added: 0047770
2020-07-03 05:24 oliverthered File Added: 2020-07-03.png
2020-07-03 05:24 oliverthered Note Added: 0047771
2020-07-03 16:05 oliverthered Note Added: 0047774
2020-07-03 17:26 jghali Note Added: 0047778
2020-07-03 17:34 oliverthered Note Added: 0047780
2020-07-03 17:36 oliverthered Note Added: 0047781
2020-07-03 17:56 oliverthered Note Added: 0047782
2020-07-03 18:42 oliverthered Note Added: 0047785
2020-07-03 20:11 oliverthered Note Added: 0047786
2020-07-03 20:25 oliverthered File Added: pdftext-import-phase1.rar
2020-07-03 20:25 oliverthered Note Added: 0047787
2020-07-06 17:35 oliverthered File Added: pdf-text-import-phase-1-with-multipage-support.rar
2020-07-06 17:35 oliverthered Note Added: 0047791
2020-07-06 17:45 oliverthered Note Added: 0047792
2020-07-06 18:54 oliverthered File Added: pdf-text-import-phase-1-with-multipage-support-2.rar
2020-07-06 18:54 oliverthered Note Added: 0047793
2020-07-08 00:03 oliverthered Note Added: 0047799
2020-07-08 00:04 oliverthered File Added: 2020-07-08.png
2020-07-11 01:05 oliverthered Note Added: 0047811
2020-07-11 01:22 oliverthered Note Added: 0047812
2020-07-11 16:56 oliverthered Note Added: 0047817
2020-07-11 16:59 oliverthered Note Added: 0047818
2020-07-11 17:35 oliverthered Note Added: 0047819
2020-07-11 17:48 oliverthered Note Added: 0047820
2020-07-11 22:11 oliverthered Note Added: 0047822
2020-07-12 00:02 jghali Note Added: 0047823
2020-07-27 13:20 jghali Note Added: 0047880
2020-10-02 17:58 mara004 Note Added: 0048123
2020-10-02 21:27 jghali Note Added: 0048124