From 89dacd8eba529663ab9c410efffd698b3023a9d6 Mon Sep 17 00:00:00 2001 From: Jay Berkenbilt Date: Sat, 17 Aug 2019 22:37:05 -0400 Subject: [PATCH] Array optimization fixes performance issues (fixes #305, #311) --- ChangeLog | 8 ++++++++ TODO | 7 ------- manual/qpdf-manual.xml | 8 ++++++++ 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/ChangeLog b/ChangeLog index 5de76343..cf51c5b3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +2019-08-17 Jay Berkenbilt + + * Change internal implementation of QPDF arrays to use sparse + arrays, which results in using much less memory for arrays with + large numbers of nulls. Various files have been encountered in the + wild that contains thousands of arrays with millions of nulls. + Fixes #305, #311. + 2019-07-03 Jay Berkenbilt * Non-compatible API change: change diff --git a/TODO b/TODO index f79b0ae4..a80fcebc 100644 --- a/TODO +++ b/TODO @@ -1,10 +1,3 @@ -Performance -=========== - -There is some performance problem that happens with some large files. -Issues 305 and 311 both seem to be examples of this. Files saved in -../misc/bugs. - Fuzz Errors =========== diff --git a/manual/qpdf-manual.xml b/manual/qpdf-manual.xml index d5ad6466..e2544cb9 100644 --- a/manual/qpdf-manual.xml +++ b/manual/qpdf-manual.xml @@ -4480,6 +4480,14 @@ print "\n"; getUIntValueAsUInt. + + + The underlying implementation of QPDF arrays has been + enhanced to be much more memory efficient when dealing with + arrays with lots of nulls. This enables qpdf to use + drastically less memory for certain types of files. + + A new helper method