View Issue Details

IDProjectCategoryView StatusLast Update
0001036ScribusUser Interfacepublic2018-05-26 10:14
ReporterringercAssigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status confirmedResolutionopen 
Platformx86 LinuxOSFedora CoreOS Version1
Product Version1.2CVS 
Target Version1.6 milestone 
Summary0001036: Severe performance problem on documents with very large number of linked frames
DescriptionThis issue was the result of testing done with a script created by the purpose. I am not aware of anybody trying to actually use Scribus for documents like those created by this script. (correction: No longer the case). I've filed it as low priority for that reason. It's filed as major as it makes the app almost unusable for documents with many linked frames and plenty of text.

When viewing very large documents containing a long series of linked frames, Scribus redraws the canvas incredibly slowly and consumes 100% CPU while doing so.

The further through the document one is viewing, the faster the redraws are. Page 220 of a 251 page document is quite usable, page 150 is getting quite painful, page 10 is almost unusable, and by page 1 Scribus may hang for minutes. Others with slower systems have reported apparent hangs even on a 43 page cut down version. If enough blank pages are inserted into the beginning of the document that the initial text frame is no longer visible when viewing the start of the document, then things are again fast and usable.

It appears that the linked frames are causing the issue. Perhaps Scribus is recalculating the text flow far more often than it needs to, and for the entire document rather than the visible parts of it.

I wrote another script that scans back through the document and unlinks the frames after saving their content, then restores the content after unlinking. After this script is run on a document, performance is back to normal and the same throughout the document.
Steps To Reproduce(1) Download attached scripts 'odyssey.py' and 'odyssey_unlink.py' as well as the file 'odyssey.txt'. Place all three into a directory and cd into it.

(2) Read the scripts to make sure they won't harm your system by incompetence or malice. Failure to do so is your problem, not mine. I assure you they're designed to be utterly harmless as can be, but can't make any guarantees on incompetence. The use_termination_hack stuff is important for another bug (coming soon) and can be left at True. Consider editing the script to create less than the full document at first though - starting at 40 pages seems sensible on a fast box, 10 on a slow one. Do not set unlink_after_create to True - that's also for the other bug.

(3) Launch scribus from the directory you saved the downloaded files into. (probably not required, but let's do it to make sure).

(4) Script -> Execute Script, choose 'odyssey.py'.

(5) In the resulting file dialog, choose 'odyssey.txt' from wherever you saved it. It's probably already visible if you followed the instructions.

(6) Wait for the script to complete. It'll inform you of what it's doing using the status bar and progress bar.

(7) Note how long it takes after the script terminates before Scribus is usable.

(8) Force a redraw of the page by dragging the properties pallette over the text and off again. Note how long this takes.

(9) Using the page pop-up menu in the bottom toolbar, switch to the last (or second last - in the full 251 page doc the last page is almost empty) page on the document.

(10) note how quickly it redraws, and how quickly you can work on it.

(11) Once you have a feel for the performance difference, etc, run the unlinker script by choosing Script -> Execute Script and running 'odyssey_unlink.py'. It will go through and copy the text out of each frame, unlink the frame, and restore the text. (It'd be nice to be able to unlink without clearing). It also adds some informational frames on what it's done, for debugging.

(12) Note that performance is now consistent and usable throughout the document.
Additional InformationTest machine uses Python 2.3.3 . Python 2.3 strongly recommended; earlier versions may not run this script correctly.
TagsNo tags attached.
PatchNo

Relationships

related to 0014946 new Suddenly start working extremely slow 
related to 0014146 new Even one text frame has serious performance issues. 
has duplicate 0006477 closedjghali Canvas refresh is very slow 
has duplicate 0006509 closed Pasting text in Story Editor is very slow 
has duplicate 0015219 closedjghali Impossible to work with large text in many linked text frames 
related to 0001037 closedsubik GetTextLength returns 0 for non-empty frame 
related to 0014032 closedjghali Severe performance problem on documents with 8 linked frames 
related to 0012825 confirmed Severe performance issue with layers having special blend modes 
child of 0003432 closed Scribus 1.3.x: low performance 
child of 0002438 acknowledged Metabug: Cases of extreme memory use 

Activities

2004-09-05 16:42

 

odyssey.py (10,866 bytes)

2004-09-05 16:42

 

odyssey_unlink.py (1,738 bytes)

2004-09-05 16:43

 

odyssey.txt.bz2 (161,971 bytes)

2004-09-05 16:47

 

odyssey.sla.bz2 (172,806 bytes)

ringerc

2004-09-05 16:48

reporter   ~0002288

You will also need to decompress odyssey.txt.bz2 to odyssey.txt. 'odyssey.sla.bz2' is a compressed version of the .sla file generated by the script - without running odyssey_unlink.py - on my system.

Frank_Cox

2004-09-27 08:01

reporter   ~0002541

This issue has now become a problem in a real-world application. I am importing a long file of classified ads (about 2500 ads or more) into 40 pages of linked text frames. After importing the file, Scribus grinds to a near-halt and it is almost impossible to update the document any further.

ringerc

2004-09-27 08:10

reporter   ~0002542

Promoting priority back to normal (I created the bug with priority low) now that someone has hit this in real work.

qp

2005-10-14 18:29

reporter   ~0007087

I'm hitting this problem in practice as well.

I've got a small book of short stories I'm trying to typeset (27,000 words). I created a new document with "Automatic Text Frames" checked. Then I imported a text file which filled the first page. Then added a load of new pages: the text automatically flowed into them.

But the screen redrawing was painfully slow. CPU goes to 100% for a few seconds on each page, every time you go to it (not just when it's filling it).

I would think this is a fairly common usage?

This is Scribus 1.2.2.99 on Debian unstable, Athlon 1.4 GHz.

ringerc

2005-10-15 04:48

reporter   ~0007091

Yep. There seem to be a lot of factors. It's less severe than it was due to general improvements in text rendering speed, but the core issues remain. Hopefully some of them will be addressed in the 1.3.x series. There's some work going on as part of the "great breakage" that should lay the groundwork for that.

2007-03-18 12:56

 

wiezasnow12-8.sla.gz (579,374 bytes)

Magnes

2007-03-18 12:57

reporter   ~0015555

Scribus 1.3.3.8 and (even more) 1.3.4 CVS (compiled in February) are VERY slow when coping with linked text frames, especially if word wrap is activated for them, even if there are only 2 linked frames. The scrolling of the page is awfully slow as is changing paragraph styles, moving frames or pictures, editing text in frames etc.
I uploaded a file (wiezasnow12-9.sla.gz - work in progress without images) where the problem occurs.

ringerc

2007-03-18 16:46

reporter   ~0015557

FYI Magnes, Andreas is doing lots of work on the text engine in 1.3.x that should result in much improved rendering speeds and results. His work is coming along very well now.

Magnes

2007-06-15 11:26

reporter   ~0016554

When the new text engine will be ready? I didn't use Scribus 1.3.4 for some time, but the last time I did it was unusable when having many linked frames. I hope to be able to use it instead of 1.3.3.9/10 the next time I'll be making some PDFs :).

mecirt

2007-07-05 10:41

reporter   ~0016745

This issue still exists on 1.3.4. Was doing a test document with 256 pages (because I usually need to do documents this big - books and such), and while repainting isn't so bad, text manipulation is very slow (64-bit turion, 1.6GHz, 1 GB RAM). Loading a text (.txt file has 300 KB) into Scribus took several minutes, and any style manipulations take 30+ seconds each.

Magnes

2007-08-26 17:09

reporter   ~0017170

Last edited: 2007-08-26 17:48

I'm not entirely sure, but it looks like Scribus 1.3.3.10CVS is slower than 1.3.3.9 when scrolling (almost as slow as 1.3.4) even with only few (3) linked text frames. I could upload test file but I don't know if it's a good bug report to do it ;).

edit: scrolling is slow when some new text (with word wrap and in linked frame) had to be drawn.

timpul

2007-08-26 19:45

reporter   ~0017171

wiezasnow12-8.sla.gz has 61 pages in xml. In xml, because my Scribus cannot open it. I think the file is so so so so so so complex to handle. I write a book with pictures and texts. I am on 80th page. the file is still 2MB. Your file must be so complex with 27MB. Huhhh.

Magnes

2007-08-26 20:18

reporter   ~0017173

Last edited: 2007-08-26 20:20

You should be able to open it if you have 1GB free memory. Although it takes forever on 1.3.4 (maybe even several minutes). It's not so complex, I thing the size is due to some additional data stored by Scribus (image miniatures?) or due to some bug...
PS. I looked into the xml file... There is a lot, lot of something like this:

<ITEXT CAB="64" CCOLOR="Black" CBASE="0" CSTW="-0.1" CSIZE="8.5" CULP="-0.1" CH="o" CSHADE2="100" CKERN="0" CSCALE="100" CSHADE="100" COUT="1" CSCALEV="100" CSTP="-0.1" CULW="-0.1" CSTYLE="128" CFONT="Georgia Regular" CSHX="5" CSHY="-5" CSTROKE="Black" />
   <ITEXT CAB="64" CCOLOR="Black" CBASE="0" CSTW="-0.1" CSIZE="8.5" CULP="-0.1" CH="nie Ch" CSHADE2="100" CKERN="0" CSCALE="100" CSHADE="100" COUT="1" CSCALEV="100" CSTP="-0.1" CULW="-0.1" CSTYLE="0" CFONT="Georgia Regular" CSHX="5" CSHY="-5" CSTROKE="Black" />
   <ITEXT CAB="64" CCOLOR="Black" CBASE="0" CSTW="-0.1" CSIZE="8.5" CULP="-0.1" CH="a" CSHADE2="100" CKERN="0" CSCALE="100" CSHADE="100" COUT="1" CSCALEV="100" CSTP="-0.1" CULW="-0.1" CSTYLE="128" CFONT="Georgia Regular" CSHX="5" CSHY="-5" CSTROKE="Black" />

Maybe it should be in one ITEXT (all options are the same)? It's really, really strange.
It could be the cause of the time it takes to open the file. (sorry for my English, I hope it's comprehensible)

ringerc

2007-08-27 00:46

reporter   ~0017177

This issue doesn't directly relate to the XML format. If Scribus is drawing content, the XML formatted content has already been loaded into RAM (entirely), converted to internal objects, and discarded from memory.

There are indeed known issues with the current format (including size, functionality limits, difficulty of direct manipulation, and the lack of any way to validate it as correct) and the team are working on a new format - as well as a lot of other things, which is why it's taking a while.

For file format issues see 0002098.

ArneBab

2009-04-08 10:08

reporter   ~0021507

I'm also getting this problem (with a 48 pages document with two text files running through 47 pages, each).

It also manifests itself when you right click in the first as opposed to the last linked frame.

JLuc

2009-12-28 22:36

developer   ~0023004

With 1.3.6svn and a midly modern computer (2Go + Turion 64x2 1.6MHz), it is quite ok to open wiezasnow12-8.sla and edit it. Editing that file is sluggish but doable : it takes about or less than one second to select or apply a style or move an image.

However, loading odyssey.sla (uncompressed from bz2) crashes scribus with a stack overflow after 60 seconds of loading and computing.
Are linked frames stack greedy ?

JLuc

2011-10-18 21:56

developer   ~0027047

tried again with 1.4.0 rc6 and almost same conclusion :
- no problem at all with

- odyssee.py doesnt crash scribus but hangs with nicely displayed following message :

Traceback (most recent call last):
  File "<string>", line 8, in <module>
  File "C:\Users\JLuc\Downloads\odyssey.py", line 67, in <module>
    scribus.Justify_Left = scribus.LeftAlign
AttributeError: 'module' object has no attribute 'LeftAlign'

ale

2011-10-19 06:40

manager   ~0027048

afaict, the constant is scribus.ALIGN_LEFT and not scribus.LeftAlign...

in 1.4, the align constants are:

ALIGN_BLOCK', 'ALIGN_CENTERED', 'ALIGN_FORCED', 'ALIGN_LEFT', 'ALIGN_RIGHT'

JLuc

2011-10-19 12:21

developer   ~0027050

I edited odissey.py file but there are numerous other mistakes, some of them i could sort out, some other i couldnt.

Issue History

Date Modified Username Field Change
2004-09-05 16:42 ringerc New Issue
2004-09-05 16:42 ringerc File Added: odyssey.py
2004-09-05 16:42 ringerc File Added: odyssey_unlink.py
2004-09-05 16:43 ringerc File Added: odyssey.txt.bz2
2004-09-05 16:47 ringerc File Added: odyssey.sla.bz2
2004-09-05 16:48 ringerc Note Added: 0002288
2004-09-27 08:01 Frank_Cox Note Added: 0002541
2004-09-27 08:10 ringerc Note Added: 0002542
2004-09-27 08:10 ringerc Priority low => normal
2004-09-27 08:10 ringerc Description Updated
2004-09-27 08:10 ringerc Steps to Reproduce Updated
2004-10-19 05:53 ringerc Relationship added related to 0001037
2004-12-08 15:59 ringerc Relationship deleted related to 0001037
2004-12-08 15:59 ringerc Relationship added related to 0001037
2005-10-14 18:29 qp Note Added: 0007087
2005-10-15 04:48 ringerc Note Added: 0007091
2006-03-15 16:01 mhanski Relationship added child of 0003432
2006-04-02 02:28 ringerc Status new => confirmed
2007-03-18 12:56 Magnes File Added: wiezasnow12-8.sla.gz
2007-03-18 12:57 Magnes Note Added: 0015555
2007-03-18 16:46 ringerc Note Added: 0015557
2007-06-15 11:26 Magnes Note Added: 0016554
2007-07-05 10:41 mecirt Note Added: 0016745
2007-08-26 17:09 Magnes Note Added: 0017170
2007-08-26 17:10 Magnes Note Edited: 0017170
2007-08-26 17:48 Magnes Note Edited: 0017170
2007-08-26 19:45 timpul Note Added: 0017171
2007-08-26 20:18 Magnes Note Added: 0017173
2007-08-26 20:20 Magnes Note Edited: 0017173
2007-08-27 00:46 ringerc Note Added: 0017177
2008-10-27 00:49 jghali Relationship added has duplicate 0006477
2009-04-08 10:08 ArneBab Note Added: 0021507
2009-12-28 22:36 JLuc Note Added: 0023004
2011-10-18 21:56 JLuc Note Added: 0027047
2011-10-19 06:40 ale Note Added: 0027048
2011-10-19 12:21 JLuc Note Added: 0027050
2014-09-18 22:06 Kunda Relationship added child of 0002438
2015-05-25 03:29 Kunda Patch => No
2015-05-25 03:29 Kunda Target Version => 1.6 milestone
2016-01-14 07:51 Kunda Relationship added related to 0006509
2016-05-06 16:20 Kunda Relationship added related to 0014032
2016-11-17 11:24 Kunda Relationship added related to 0012825
2016-11-17 11:36 Kunda Relationship replaced has duplicate 0006509
2018-03-25 19:35 jghali Relationship added has duplicate 0015219
2018-05-26 10:14 PeterBenedek Relationship added related to 0014946
2018-05-26 10:14 PeterBenedek Relationship added related to 0014146