mirror of
https://github.com/qpdf/qpdf.git
synced 2024-06-02 02:10:52 +00:00
Merge pull request #998 from m-holger/todo
Add content table to TODO file
This commit is contained in:
commit
b8f20fe34c
|
@ -15,3 +15,4 @@ d740c6ccced02147f84a39d5e5f0984d12bac6cb
|
|||
# Reflow comments and strings to 100 columns
|
||||
698a70e6a84cf7c0db667e9d9e021b4c34c85a3e
|
||||
3c5700c255f4603b5df9c6d183d13dd71a083cc3
|
||||
9ae7bdea966102f9621b22192747a891078e7470
|
||||
|
|
844
TODO
844
TODO
|
@ -1,844 +0,0 @@
|
|||
Always
|
||||
======
|
||||
|
||||
* Evaluate issues tagged with `next` and `bug`. Remember to check
|
||||
discussions and pull requests in addition to regular issues.
|
||||
* When close to release, make sure external-libs is building and
|
||||
follow instructions in ../external-libs/README
|
||||
|
||||
Next
|
||||
====
|
||||
|
||||
* Fix #874 -- make args in --encrypt to match the json and make
|
||||
positional fill in the gaps
|
||||
* Maybe fix #553 -- use file times for attachments
|
||||
* std::string_view transition -- work being done by m-holger
|
||||
* Break ground on "Document-level work" -- TODO-pages.md lives on a
|
||||
separate branch.
|
||||
* Standard for CLI and Job JSON support for JSON-based command-line
|
||||
arguments. Come up with a standard way of supporting command-line
|
||||
arguments that take JSON specifications of things so that
|
||||
* there is a predictable way to indicate whether an argument is a
|
||||
file or a JSON blob
|
||||
* with QPDFJob JSON, make sure it is possible to directly include
|
||||
the JSON rather than having to stringify a JSON blob
|
||||
* One option might be to prepend file:// to a filename or otherwise
|
||||
to take a JSON blob. We could have that as a particular type of
|
||||
argument that would behave properly for both job JSON and CLI.
|
||||
|
||||
|
||||
Possible future JSON enhancements
|
||||
=================================
|
||||
|
||||
* Consider not including unreferenced objects and trimming the trailer
|
||||
in the same way that QPDFWriter does (except don't remove `/ID`).
|
||||
This means excluding the linearization dictionary and hint stream,
|
||||
the encryption dictionary, all keys from trailer that are removed by
|
||||
QPDFWriter::getTrimmedTrailer except `/ID`, any object streams, and
|
||||
the xref stream as long as all those objects are unreferenced. (They
|
||||
always should be, but there could be some bizarre case of someone
|
||||
creating a PDF file that has an indirect reference to one of those,
|
||||
in which case we need to preserve it.) If this is done, make
|
||||
`--preserve-unreferenced` preserve unreference objects and also
|
||||
those extra keys. Search for "linear" and "trailer" in json.rst to
|
||||
update the various places in the documentation that discuss this.
|
||||
Also update the help for --json and --preserve-unreferenced.
|
||||
|
||||
* Add to JSON output the information available from a few additional
|
||||
informational options:
|
||||
|
||||
* --check: add but maybe not by default?
|
||||
|
||||
* --show-linearization: add but maybe not by default? Also figure
|
||||
out whether warnings reported for some of the PDF specs (1.7) are
|
||||
qpdf problems. This may not be worth adding in the first
|
||||
increment.
|
||||
|
||||
* --show-xref: add
|
||||
|
||||
* Consider having --check, --show-encryption, etc., just select the
|
||||
right keys when in json mode. I don't think I want check on by
|
||||
default, so that might be different.
|
||||
|
||||
* Consider having warnings be included in the json in a "warnings" key
|
||||
in json mode.
|
||||
|
||||
QPDFJob
|
||||
=======
|
||||
|
||||
Here are some ideas for QPDFJob that didn't make it into 10.6. Not all
|
||||
of these are necessarily good -- just things to consider.
|
||||
|
||||
* How do we chain jobs? The idea would be that the input and/or output
|
||||
of a QPDFJob could be a QPDF object rather than a file. For input,
|
||||
it's pretty easy. For output, none of the output-specific options
|
||||
(encrypt, compress-streams, objects-streams, etc.) would have any
|
||||
affect, so we would have to treat this like inspect for error
|
||||
checking. The QPDF object in the state where it's ready to be sent
|
||||
off to QPDFWriter would be used as the input to the next QPDFJob.
|
||||
For the job json, I think we can have the output be an identifier
|
||||
that can be used as the input for another QPDFJob. For a json file,
|
||||
we could the top level detect if it's an array with the convention
|
||||
that exactly one has an output, or we could have a subkey with other
|
||||
job definitions or something. Ideally, any input
|
||||
(copy-attachments-from, pages, etc.) could use a QPDF object. It
|
||||
wouldn't surprise me if this exposes bugs in qpdf around foreign
|
||||
streams as this has been a relatively fragile area before.
|
||||
|
||||
Documentation
|
||||
=============
|
||||
|
||||
* Do a full pass through the documentation.
|
||||
|
||||
* Make sure `qpdf` is consistent. Use QPDF when just referring to
|
||||
the package.
|
||||
* Make sure markup is consistent
|
||||
* Autogenerate where possible
|
||||
* Consider which parts might be good candidates for moving to the
|
||||
wiki.
|
||||
|
||||
* Commit 'Manual - enable line wrapping in table cells' from
|
||||
Mon Jan 17 12:22:35 2022 +0000 enables table cell wrapping. See if
|
||||
this can be incorporated directly into sphinx_rtd_theme and the
|
||||
workaround can be removed.
|
||||
|
||||
* When possible, update the debian package to include docs again. See
|
||||
https://bugs.debian.org/1004159 for details.
|
||||
|
||||
Document-level work
|
||||
===================
|
||||
|
||||
* Ideas here may by superseded by #593.
|
||||
|
||||
* QPDFPageCopier -- object for moving pages around within files or
|
||||
between files and performing various transformations. Reread/rewrite
|
||||
_page-selection in the manual if needed.
|
||||
|
||||
* Handle all the stuff of pages and split-pages
|
||||
* Do n-up, booklet, collation
|
||||
* Look through cli and see what else...flatten-*?
|
||||
* See comments in QPDFPageDocumentHelper.hh for addPage -- search
|
||||
for "a future version".
|
||||
* Make it efficient for bulk operations
|
||||
* Make certain doc-level features selectable
|
||||
* qpdf.cc should do all its page operations, including
|
||||
overlay/underlay, splitting, and merging, using this
|
||||
* There should also be example code
|
||||
|
||||
* After doc-level checks are in, call --check on the output files in
|
||||
the "Copy Annotations" tests.
|
||||
|
||||
* Document-level checks. For example, for forms, make sure all form
|
||||
fields point to an annotation on exactly one page as well as that
|
||||
all widget annotations are associated with a form field. Hook this
|
||||
into QPDFPageCopier as well as the doc helpers. Make sure it is
|
||||
called from --check.
|
||||
|
||||
* See also issues tagged with "pages". Include closed issues.
|
||||
|
||||
* Add flags to CLI to select which document-level options to
|
||||
preserve or not preserve. We will probably need a pair of mutually
|
||||
exclusive, repeatable options with a way to specify all, none, only
|
||||
{x,y}, or all but {x,y}.
|
||||
|
||||
* If a page contains a reference a file attachment annotation, when
|
||||
that page is copied, if the file attachment appears in the top-level
|
||||
EmbeddedFiles tree, that entry should be preserved in the
|
||||
destination file. Otherwise, we probably will require the use of
|
||||
--copy-attachments-from to preserve these. What will the strategy be
|
||||
for deduplicating in the automatic case?
|
||||
|
||||
Text Appearance Streams
|
||||
=======================
|
||||
|
||||
This is a list of known issues with text appearance streams and things
|
||||
we might do about it.
|
||||
|
||||
* For variable text, the spec says to pull any resources from /DR that
|
||||
are referenced in /DA but if the resource dictionary already has
|
||||
that resource, just use the one that's there. The current code looks
|
||||
only for /Tf and adds it if needed. We might want to instead merge
|
||||
/DR with resources and then remove anything that's unreferenced. We
|
||||
have all the code required for that in ResourceFinder except
|
||||
TfFinder also gets the font size, which ResourceFinder doesn't do.
|
||||
|
||||
* There are things we are missing because we don't look at font
|
||||
metrics. The code from TextBuilder (work) has almost everything in
|
||||
it that is required. Once we have knowledge of character widths, we
|
||||
can support quadding and multiline text fields (/Ff 4096), and we
|
||||
can potentially squeeze text to fit into a field. For multiline,
|
||||
first squeeze vertically down to the font height, then squeeze
|
||||
horizontally with Tz. For single line, squeeze horizontally with Tz.
|
||||
If we use Tz, issue a warning.
|
||||
|
||||
* When mapping characters to widths, we will need to care about
|
||||
character encoding. For built-in fonts, we can create a map from
|
||||
Unicode code point to width and then go from the font's encoding to
|
||||
unicode to the width. See misc/character-encoding/ (not on github)
|
||||
and font metric information for the 14 standard fonts in my local
|
||||
pdf-spec directory.
|
||||
|
||||
* Once we know about character widths, we can correctly support
|
||||
auto-sized variable text fields (0 Tf). If this is fixed, search for
|
||||
"auto-sized" in cli.rst.
|
||||
|
||||
Fuzz Errors
|
||||
===========
|
||||
|
||||
* https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=<N>
|
||||
|
||||
* Ignoring these:
|
||||
* Out of memory in dct: 35001, 32516
|
||||
|
||||
External Libraries
|
||||
==================
|
||||
|
||||
Current state (10.0.2):
|
||||
|
||||
* qpdf/external-libs repository builds external-libs on a schedule.
|
||||
It detects and downloads the latest versions of zlib, jpeg, and
|
||||
openssl and creates source and binary distribution zip files in an
|
||||
artifact called "distribution".
|
||||
|
||||
* Releases in qpdf/external-libs are made manually. They contain
|
||||
qpdf-external-libs-{bin,src}.zip.
|
||||
|
||||
* The qpdf build finds the latest non-prerelease release and downloads
|
||||
the qpdf-external-libs-*.zip files from the releases in the setup
|
||||
stage.
|
||||
|
||||
* To upgrade to a new version of external-libs, create a new release
|
||||
of qpdf/external-libs (see README-maintainer in external-libs) from
|
||||
the distribution artifact of the most recent successful build after
|
||||
ensuring that it works.
|
||||
|
||||
Desired state:
|
||||
|
||||
* The qpdf/external-libs repository should create release candidates.
|
||||
Ideally, every scheduled run would make its zip files available. A
|
||||
personal access token with actions:read scope for the
|
||||
qpdf/external-libs repository is required to download the artifact
|
||||
from an action run, and qpdf/qpdf's secrets.GITHUB_TOKEN doesn't
|
||||
have this access. We could create a service account for this
|
||||
purpose. As an alternative, we could have a draft release in
|
||||
qpdf/external-libs that the qpdf/external-libs build could update
|
||||
with each candidate. It may also be possible to solve this by
|
||||
developing a simple GitHub app.
|
||||
|
||||
* Scheduled runs of the qpdf build in the qpdf/qpdf repository (not a
|
||||
fork or pull request) could download external-libs from the release
|
||||
candidate area instead of the latest stable release. Pushes to the
|
||||
build branch should still use the latest release so it always
|
||||
matches the main branch.
|
||||
|
||||
* Periodically, we would create a release of external-libs from the
|
||||
release candidate zip files. This could be done safely because we
|
||||
know the latest qpdf works with it. This could be done at least
|
||||
before every release of qpdf, but potentially it could be done at
|
||||
other times, such as when a new dependency version is available or
|
||||
after some period of time.
|
||||
|
||||
Other notes:
|
||||
|
||||
* The external-libs branch in qpdf/qpdf was never documented. We might
|
||||
be able to get away with deleting it.
|
||||
|
||||
* See README-maintainer in qpdf/external-libs for information on
|
||||
creating a release. This could be at least partially scripted in a
|
||||
way that works for the qpdf/qpdf repository as well since they are
|
||||
very similar.
|
||||
|
||||
ABI Changes
|
||||
===========
|
||||
|
||||
This is a list of changes to make next time there is an ABI change.
|
||||
Comments appear in the code prefixed by "ABI".
|
||||
|
||||
Always:
|
||||
* Search for ABI in source and header files
|
||||
* Search for "[[deprecated" to find deprecated APIs that can be removed
|
||||
* Search for issues, pull requests, and discussions with the "abi" label
|
||||
* Check discussion "qpdf X planning" where X is the next major
|
||||
version. This should be tagged `abi`
|
||||
|
||||
For qpdf 12, see https://github.com/qpdf/qpdf/discussions/785
|
||||
|
||||
C++ Version Changes
|
||||
===================
|
||||
|
||||
Use
|
||||
// C++NN: ...
|
||||
to mark places in the code that should be updated when we require at
|
||||
least that version of C++.
|
||||
|
||||
Page splitting/merging
|
||||
======================
|
||||
|
||||
* Update page splitting and merging to handle document-level
|
||||
constructs with page impact such as interactive forms and article
|
||||
threading. Check keys in the document catalog for others, such as
|
||||
outlines, page labels, thumbnails, and zones. For threads,
|
||||
Subramanyam provided a test file; see ../misc/article-threads.pdf.
|
||||
Email Q-Count: 431864 from 2009-11-03.
|
||||
|
||||
* bookmarks (outlines) 12.3.3
|
||||
* support bookmarks when merging
|
||||
* prune bookmarks that don't point to a surviving page when merging
|
||||
or splitting
|
||||
* make sure conflicting named destinations work possibly test by
|
||||
including the same file by two paths in a merge
|
||||
* see also comments in issue 343
|
||||
|
||||
Note: original implementation of bookmark preservation for split
|
||||
pages caused a very high performance hit. The problem was
|
||||
introduced in 313ba081265f69ac9a0324f9fe87087c72918191 and reverted
|
||||
in the commit that adds this paragraph. The revert includes marking
|
||||
a few tests cases as $td->EXPECT_FAILURE. When properly coded, the
|
||||
test cases will need to be adjusted to only include the parts of
|
||||
the outlines that are actually copied. The tests in question are
|
||||
"split page with outlines". When implementing properly, ensure that
|
||||
the performance is not adversely affected by timing split-pages on
|
||||
a large file with complex outlines such as the PDF specification.
|
||||
|
||||
When pruning outlines, keep all outlines in the hierarchy that are
|
||||
above an outline for a page we care about. If one of the ancestor
|
||||
outlines points to a non-existent page, clear its dest. If an
|
||||
outline does not have any children that point to pages in the
|
||||
document, just omit it.
|
||||
|
||||
Possible strategy:
|
||||
* resolve all named destinations to explicit destinations
|
||||
* concatenate top-level outlines
|
||||
* prune outlines whose dests don't point to a valid page
|
||||
* recompute all /Count fields
|
||||
|
||||
Test files
|
||||
* page-labels-and-outlines.pdf: old file with both page labels and
|
||||
outlines. All destinations are explicit destinations. Each page
|
||||
has Potato and a number. All titles are feline names.
|
||||
* outlines-with-actions.pdf: mixture of explicit destinations,
|
||||
named destinations, goto actions with explicit destinations, and
|
||||
goto actions with named destinations; uses /Dests key in names
|
||||
dictionary. Each page has Salad and a number. All titles are
|
||||
silly words. One destination is an indirect object.
|
||||
* outlines-with-old-root-dests.pdf: like outlines-with-actions
|
||||
except it uses the PDF-1.1 /Dests dictionary for named
|
||||
destinations, and each page has Soup and a number. Also pages are
|
||||
numbered with upper-case Roman numerals starting with 0. All
|
||||
titles are silly words preceded by a bullet.
|
||||
|
||||
If outline handling is significantly improved, see
|
||||
../misc/bad-outlines/bad-outlines.pdf and email:
|
||||
https://mail.google.com/mail/u/0/#search/rfc822msgid%3A02aa01d3d013%249f766990%24de633cb0%24%40mono.hr)
|
||||
|
||||
* Form fields: should be similar to outlines.
|
||||
|
||||
Analytics
|
||||
=========
|
||||
|
||||
Consider features that make it easier to detect certain patterns in
|
||||
PDF files. The information below could be computed using an external
|
||||
program that reads the existing json, but if it's useful enough, we
|
||||
could add it directly to the json output.
|
||||
|
||||
* Add to "pages" in the json:
|
||||
* "inheritsresources": bool; whether there are any inherited
|
||||
attributes from ancestor page tree nodes
|
||||
* "sharedresources": a list of indirect objects that are
|
||||
"/Resources" dictionaries or "XObject" resource dictionary subkeys
|
||||
of either the page itself or of any form XObject referenced by the
|
||||
page.
|
||||
|
||||
* Add to "objectinfo" in json: "directpagerefcount": the number of
|
||||
pages that directly reference this object (i.e., you can find an
|
||||
indirect reference to the object in the page dictionary without
|
||||
traversing over any indirect objects)
|
||||
|
||||
General
|
||||
=======
|
||||
|
||||
NOTE: Some items in this list refer to files in my personal home
|
||||
directory or that are otherwise not publicly accessible. This includes
|
||||
things sent to me by email that are specifically not public. Even so,
|
||||
I find it useful to make reference to them in this list.
|
||||
|
||||
* Consider enabling code scanning on GitHub.
|
||||
|
||||
* Add an option --ignore-encryption to ignore encryption information
|
||||
and treat encrypted files as if they weren't encrypted. This should
|
||||
make it possible to solve #598 (--show-encryption without a
|
||||
password). We'll need to make sure we don't try to filter any
|
||||
streams in this mode. Ideally we should be able to combine this with
|
||||
--json so we can look at the raw encrypted strings and streams if we
|
||||
want to, though be sure to document that the resulting JSON won't be
|
||||
convertible back to a valid PDF. Since providing the password may
|
||||
reveal additional details, --show-encryption could potentially retry
|
||||
with this option if the first time doesn't work. Then, with the file
|
||||
open, we can read the encryption dictionary normally. If this is
|
||||
done, search for "raw, encrypted" in json.rst.
|
||||
|
||||
* In libtests, separate executables that need the object library
|
||||
from those that strictly use public API. Move as many of the test
|
||||
drivers from the qpdf directory into the latter category as long
|
||||
as doing so isn't too troublesome from a coverage standpoint.
|
||||
|
||||
* Consider generating a non-flat pages tree before creating output to
|
||||
better handle files with lots of pages. If there are more than 256
|
||||
pages, add a second layer with the second layer nodes having no more
|
||||
than 256 nodes and being as evenly sizes as possible. Don't worry
|
||||
about the case of more than 65,536 pages. If the top node has more
|
||||
than 256 children, we'll live with it. This is only safe if all
|
||||
intermediate page nodes have only /Kids, /Parent, /Type, and /Count.
|
||||
|
||||
* Look at https://bestpractices.coreinfrastructure.org/en
|
||||
|
||||
* Consider adding fuzzer code for JSON
|
||||
|
||||
* Rework tests so that nothing is written into the source directory.
|
||||
Ideally then the entire build could be done with a read-only
|
||||
source tree.
|
||||
|
||||
* Large file tests fail with linux32 before and after cmake. This was
|
||||
first noticed after 10.6.3. I don't think it's worth fixing.
|
||||
|
||||
* Consider updating the fuzzer with code that exercises
|
||||
copyAnnotations, file attachments, and name and number trees. Check
|
||||
fuzzer coverage.
|
||||
|
||||
* Add code for creation of a file attachment annotation. It should
|
||||
also be possible to create a widget annotation and a form field.
|
||||
Update the pdf-attach-file.cc example with new APIs when ready.
|
||||
|
||||
* Flattening of form XObjects seems like something that would be
|
||||
useful in the library. We are seeing more cases of completely valid
|
||||
PDF files with form XObjects that cause problems in other software.
|
||||
Flattening of form XObjects could be a useful way to work around
|
||||
those issues or to prepare files for additional processing, making
|
||||
it possible for users of the qpdf library to not be concerned about
|
||||
form XObjects. This could be done recursively; i.e., we could have a
|
||||
method to embed a form XObject into whatever contains it, whether
|
||||
that is a form XObject or a page. This would require more
|
||||
significant interpretation of the content stream. We would need a
|
||||
test file in which the placement of the form XObject has to be in
|
||||
the right place, e.g., the form XObject partially obscures earlier
|
||||
code and is partially obscured by later code. Keys in the resource
|
||||
dictionary may need to be changed -- create test cases with lots of
|
||||
duplicated/overlapping keys.
|
||||
|
||||
* Part of closed_file_input_source.cc is disabled on Windows because
|
||||
of odd failures. It might be worth investigating so we can fully
|
||||
exercise this in the test suite. That said, ClosedFileInputSource
|
||||
is exercised elsewhere in qpdf's test suite, so this is not that
|
||||
pressing.
|
||||
|
||||
* If possible, consider adding CCITT3, CCITT4, or any other easy
|
||||
filters. For some reference code that we probably can't use but may
|
||||
be handy anyway, see
|
||||
http://partners.adobe.com/public/developer/ps/sdk/index_archive.html
|
||||
|
||||
* If possible, support the following types of broken files:
|
||||
|
||||
- Files that have no whitespace token after "endobj" such that
|
||||
endobj collides with the start of the next object
|
||||
|
||||
- See ../misc/broken-files
|
||||
|
||||
- See ../misc/bad-files-issue-476. This directory contains a
|
||||
snapshot of the google doc and linked PDF files from issue #476.
|
||||
Please see the issue for details.
|
||||
|
||||
* Additional form features
|
||||
* set value from CLI? Specify title, and provide way to
|
||||
disambiguate, probably by giving objgen of field
|
||||
|
||||
* Pl_TIFFPredictor is pretty slow.
|
||||
|
||||
* Support for handling file names with Unicode characters in Windows
|
||||
is incomplete. qpdf seems to support them okay from a functionality
|
||||
standpoint, and the right thing happens if you pass in UTF-8
|
||||
encoded filenames to QPDF library routines in Windows (they are
|
||||
converted internally to wchar_t*), but file names are encoded in
|
||||
UTF-8 on output, which doesn't produce nice error messages or
|
||||
output on Windows in some cases.
|
||||
|
||||
* If we ever wanted to do anything more with character encoding, see
|
||||
../misc/character-encoding/, which includes machine-readable dump
|
||||
of table D.2 in the ISO-32000 PDF spec. This shows the mapping
|
||||
between Unicode, StandardEncoding, WinAnsiEncoding,
|
||||
MacRomanEncoding, and PDFDocEncoding.
|
||||
|
||||
* Some test cases on bad files fail because qpdf is unable to find
|
||||
the root dictionary when it fails to read the trailer. Recovery
|
||||
could find the root dictionary and even the info dictionary in
|
||||
other ways. In particular, issue-202.pdf can be opened by evince,
|
||||
and there's no real reason that qpdf couldn't be made to be able to
|
||||
recover that file as well.
|
||||
|
||||
* Audit every place where qpdf allocates memory to see whether there
|
||||
are cases where malicious inputs could cause qpdf to attempt to
|
||||
grab very large amounts of memory. Certainly there are cases like
|
||||
this, such as if a very highly compressed, very large image stream
|
||||
is requested in a buffer. Hopefully normal input to output
|
||||
filtering doesn't ever try to do this. QPDFWriter should be checked
|
||||
carefully too. See also bugs/private/from-email-663916/
|
||||
|
||||
* Interactive form modification:
|
||||
https://github.com/qpdf/qpdf/issues/213 contains a good discussion
|
||||
of some ideas for adding methods to modify annotations and form
|
||||
fields if we want to make it easier to support modifications to
|
||||
interactive forms. Some of the ideas have been implemented, and
|
||||
some of the probably never will be implemented, but it's worth a
|
||||
read if there is an intention to work on this. In the issue, search
|
||||
for "Regarding write functionality", and read that comment and the
|
||||
responses to it.
|
||||
|
||||
* Look at ~/Q/pdf-collection/forms-from-appian/
|
||||
|
||||
* When decrypting files with /R=6, hash_V5 is called more than once
|
||||
with the same inputs. Caching the results or refactoring to reduce
|
||||
the number of identical calls could improve performance for
|
||||
workloads that involve processing large numbers of small files.
|
||||
|
||||
* Consider adding a method to balance the pages tree. It would call
|
||||
pushInheritedAttributesToPage, construct a pages tree from scratch,
|
||||
and replace the /Pages key of the root dictionary with the new
|
||||
tree.
|
||||
|
||||
* Study what's required to support savable forms that can be saved by
|
||||
Adobe Reader. Does this require actually signing the document with
|
||||
an Adobe private key? Search for "Digital signatures" in the PDF
|
||||
spec, and look at ~/Q/pdf-collection/form-with-full-save.pdf, which
|
||||
came from Adobe's example site. See also
|
||||
../misc/digital-sign-from-trueroad/ and
|
||||
../misc/digital-signatures/digitally-signed-pdf-xfa.pdf. If digital
|
||||
signatures are implemented, update the docs on crypto providers,
|
||||
which mention that this may happen in the future.
|
||||
|
||||
* Qpdf does not honor /EFF when adding new file attachments. When it
|
||||
encrypts, it never generates streams with explicit crypt filters.
|
||||
Prior to 10.2, there was an incorrect attempt to treat /EFF as a
|
||||
default value for decrypting file attachment streams, but it is not
|
||||
supposed to mean that. Instead, it is intended for conforming
|
||||
writers to obey this when adding new attachments. Qpdf is not a
|
||||
conforming writer in that respect.
|
||||
|
||||
* The whole xref handling code in the QPDF object allows the same
|
||||
object with more than one generation to coexist, but a lot of logic
|
||||
assumes this isn't the case. Anything that creates mappings only
|
||||
with the object number and not the generation is this way,
|
||||
including most of the interaction between QPDFWriter and QPDF. If
|
||||
we wanted to allow the same object with more than one generation to
|
||||
coexist, which I'm not sure is allowed, we could fix this by
|
||||
changing xref_table. Alternatively, we could detect and disallow
|
||||
that case. In fact, it appears that Adobe reader and other PDF
|
||||
viewing software silently ignores objects of this type, so this is
|
||||
probably not a big deal.
|
||||
|
||||
* From a suggestion in bug 3152169, consider having an option to
|
||||
re-encode inline images with an ASCII encoding.
|
||||
|
||||
* From github issue 2, provide more in-depth output for examining
|
||||
hint stream contents. Consider adding on option to provide a
|
||||
human-readable dump of linearization hint tables. This should
|
||||
include improving the 'overflow reading bit stream' message as
|
||||
reported in issue #2. There are multiple calls to stopOnError in
|
||||
the linearization checking code. Ideally, these should not
|
||||
terminate checking. It would require re-acquiring an understanding
|
||||
of all that code to make the checks more robust. In particular,
|
||||
it's hard to look at the code and quickly determine what is a true
|
||||
logic error and what could happen because of malformed user input.
|
||||
See also ../misc/linearization-errors.
|
||||
|
||||
* If I ever decide to make appearance stream-generation aware of
|
||||
fonts or font metrics, see email from Tobias with Message-ID
|
||||
<5C3C9C6C.8000102@thax.hardliners.org> dated 2019-01-14.
|
||||
|
||||
* Look at places in the code where object traversal is being done and,
|
||||
where possible, try to avoid it entirely or at least avoid ever
|
||||
traversing the same objects multiple times.
|
||||
|
||||
----------------------------------------------------------------------
|
||||
|
||||
HISTORICAL NOTES
|
||||
|
||||
Performance
|
||||
===========
|
||||
|
||||
As described in https://github.com/qpdf/qpdf/issues/401, there was
|
||||
great performance degradation between qpdf 7.1.1 and 9.1.1. Doing a
|
||||
bisect between dac65a21fb4fa5f871e31c314280b75adde89a6c and
|
||||
release-qpdf-7.1.1, I found several commits that damaged performance.
|
||||
I fixed some of them to improve performance by about 70% (as measured
|
||||
by saying that old times were 170% of new times). The remaining
|
||||
commits that broke performance either can't be correct because they
|
||||
would re-introduce an old bug or aren't worth correcting because of
|
||||
the high value they offer relative to a relatively low penalty. For
|
||||
historical reference, here are the commits. The numbers are the time
|
||||
in seconds on the machine I happened to be using of splitting the
|
||||
first 100 pages of PDF32000_2008.pdf 20 times and taking an average
|
||||
duration.
|
||||
|
||||
Commits that broke performance:
|
||||
|
||||
* d0e99f195a987c483bbb6c5449cf39bee34e08a1 -- object description and
|
||||
context: 0.39 -> 0.45
|
||||
* a01359189b32c60c2d55b039f7aefd6c3ce0ebde (minus 313ba08) -- fix
|
||||
dangling references: 0.55 -> 0.6
|
||||
* e5f504b6c5dc34337cc0b316b4a7b1fca7e614b1 -- sparse array: 0.6 -> 0.62
|
||||
|
||||
Other intermediate steps that were previously fixed:
|
||||
|
||||
* 313ba081265f69ac9a0324f9fe87087c72918191 -- copy outlines into
|
||||
split: 0.55 -> 4.0
|
||||
* a01359189b32c60c2d55b039f7aefd6c3ce0ebde -- fix dangling references:
|
||||
4.0 -> 9.0
|
||||
|
||||
This commit fixed the awful problem introduced in 313ba081:
|
||||
|
||||
* a5a016cdd26a8e5c99e5f019bc30d1bdf6c050a2 -- revert outline
|
||||
preservation: 9.0 -> 0.6
|
||||
|
||||
Note that the fix dangling references commit had a much worse impact
|
||||
prior to removing the outline preservation, so I also measured its
|
||||
impact in isolation.
|
||||
|
||||
A few important lessons (in README-maintainer)
|
||||
|
||||
* Indirection through PointerHolder<Members> is expensive, and should
|
||||
not be used for things that are created and destroyed frequently
|
||||
such as QPDFObjectHandle and QPDFObject.
|
||||
* Traversal of objects is expensive and should be avoided where
|
||||
possible.
|
||||
|
||||
Also, it turns out that PointerHolder is more performant than
|
||||
std::shared_ptr. (This was true at the time but subsequent
|
||||
implementations of std::shared_ptr became much more efficient.)
|
||||
|
||||
QPDFPagesTree
|
||||
=============
|
||||
|
||||
On a few occasions, I have considered implementing a QPDFPagesTree
|
||||
object that would allow the document's original page tree structure to
|
||||
be preserved. See comments at the top QPDF_pages.cc for why this was
|
||||
abandoned.
|
||||
|
||||
Partial work is in refs/attic/QPDFPagesTree. QPDFPageTree is mostly
|
||||
implemented and mostly tested. There are not enough cases of different
|
||||
kinds of operations (pclm, linearize, json, etc.) with non-flat pages
|
||||
trees. Insertion is not implemented. Insertion is potentially complex
|
||||
because of the issue of inherited objects. We will have to call
|
||||
pushInheritedAttributesToPage before adding any pages to the pages
|
||||
tree. The test suite is failing on that branch.
|
||||
|
||||
Some parts of page tree repair are silent (no warnings). All page tree
|
||||
repair should warn. The reason is that page tree repair will change
|
||||
object numbers, and knowing that is important when working with JSON
|
||||
output.
|
||||
|
||||
If we were to do this, we would still need keep a pages cache for
|
||||
efficient insertion. There's no reason we can't keep a vector of page
|
||||
objects up to date and just do a traversal the first time we do
|
||||
getAllPages just like we do now. The difference is that we would not
|
||||
flatten the pages tree. It would be useful to go through QPDF_pages
|
||||
and reimplement everything without calling flattenPagesTree. Then we
|
||||
can remove flattenPagesTree, which is private. That said, with the
|
||||
addition of creating non-flat pages trees, there is really no reason
|
||||
not to flatten the pages tree for internal use.
|
||||
|
||||
In its current state, QPDFPagesTree does not proactively fix /Type or
|
||||
correct page objects that are used multiple times. You have to
|
||||
traverse the pages tree to trigger this operation. It would be nice if
|
||||
we would do that somewhere but not do it more often than necessary so
|
||||
isPagesObject and isPageObject are reliable and can be made more
|
||||
reliable. Maybe add a validate or repair function? It should also make
|
||||
sure /Count and /Parent are correct.
|
||||
|
||||
Rejected Ideas
|
||||
==============
|
||||
|
||||
* Investigate whether there is a way to automate the memory checker
|
||||
tests for Windows.
|
||||
|
||||
* Provide support in QPDFWriter for writing incremental updates.
|
||||
Provide support in qpdf for preserving incremental updates. The
|
||||
goal should be that QDF mode should be fully functional for files
|
||||
with incremental updates including fix_qdf.
|
||||
|
||||
Note that there's nothing that says an indirect object in one
|
||||
update can't refer to an object that doesn't appear until a later
|
||||
update. This means that QPDF has to treat indirect null objects
|
||||
differently from how it does now. QPDF drops indirect null objects
|
||||
that appear as members of arrays or dictionaries. For arrays, it's
|
||||
handled in QPDFWriter where we make indirect nulls direct. This is
|
||||
in a single if block, and nothing else in the code cares about it.
|
||||
We could just remove that if block and not break anything except a
|
||||
few test cases that exercise the current behavior. For
|
||||
dictionaries, it's more complicated. In this case,
|
||||
QPDF_Dictionary::getKeys() ignores all keys with null values, and
|
||||
hasKey() returns false for keys that have null values. We would
|
||||
probably want to make QPDF_Dictionary able to handle the special
|
||||
case of keys that are indirect nulls and basically never have it
|
||||
drop any keys that are indirect objects.
|
||||
|
||||
If we make a change to have qpdf preserve indirect references to
|
||||
null objects, we have to note this in ChangeLog and in the release
|
||||
notes since this will change output files. We did this before when
|
||||
we stopped flattening scalar references, so this is probably not a
|
||||
big deal. We also have to make sure that the testing for this
|
||||
handles non-trivial cases of the targets of indirect nulls being
|
||||
replaced by real objects in an update. I'm not sure how this plays
|
||||
with linearization, if at all. For cases where incremental updates
|
||||
are not being preserved as incremental updates and where the data
|
||||
is being folded in (as is always the case with qpdf now), none of
|
||||
this should make any difference in the actual semantics of the
|
||||
files.
|
||||
|
||||
* The second xref stream for linearized files has to be padded only
|
||||
because we need file_size as computed in pass 1 to be accurate. If
|
||||
we were not allowing writing to a pipe, we could seek back to the
|
||||
beginning and fill in the value of /L in the linearization
|
||||
dictionary as an optimization to alleviate the need for this
|
||||
padding. Doing so would require us to pad the /L value
|
||||
individually and also to save the file descriptor and determine
|
||||
whether it's seekable. This is probably not worth bothering with.
|
||||
|
||||
* Based on an idea suggested by user "Atom Smasher", consider
|
||||
providing some mechanism to recover earlier versions of a file
|
||||
embedded prior to appended sections.
|
||||
|
||||
* Consider creating a sanitizer to make it easier for people to send
|
||||
broken files. Now that we have json mode, this is probably no
|
||||
longer worth doing. Here is the previous idea, possibly implemented
|
||||
by making it possible to run the lexer (tokenizer) over a whole
|
||||
file. Make it possible to replace all strings in a file lexically
|
||||
even on badly broken files. Ideally this should work files that are
|
||||
lacking xref, have broken links, duplicated dictionary keys, syntax
|
||||
errors, etc., and ideally it should work with encrypted files if
|
||||
possible. This should go through the streams and strings and
|
||||
replace them with fixed or random characters, preferably, but not
|
||||
necessarily, in a manner that works with fonts. One possibility
|
||||
would be to detect whether a string contains characters with normal
|
||||
encoding, and if so, use 0x41. If the string uses character maps,
|
||||
use 0x01. The output should otherwise be unrelated to the input.
|
||||
This could be built after the filtering and tokenizer rewrite and
|
||||
should be done in a manner that takes advantage of the other
|
||||
lexical features. This sanitizer should also clear metadata and
|
||||
replace images. If I ever do this, the file from issue #494 would
|
||||
be a great one to look at.
|
||||
|
||||
* Here are some notes about having stream data providers modify
|
||||
stream dictionaries. I had wanted to add this functionality to make
|
||||
it more efficient to create stream data providers that may
|
||||
dynamically decide what kind of filters to use and that may end up
|
||||
modifying the dictionary conditionally depending on the original
|
||||
stream data. Ultimately I decided not to implement this feature.
|
||||
This paragraph describes why.
|
||||
|
||||
* When writing, the way objects are placed into the queue for
|
||||
writing strongly precludes creation of any new indirect objects,
|
||||
or even changing which indirect objects are referenced from which
|
||||
other objects, because we sometimes write as we are traversing
|
||||
and enqueuing objects. For non-linearized files, there is a risk
|
||||
that an indirect object that used to be referenced would no
|
||||
longer be referenced, and whether it was already written to the
|
||||
output file would be based on an accident of where it was
|
||||
encountered when traversing the object structure. For linearized
|
||||
files, the situation is considerably worse. We decide which
|
||||
section of the file to write an object to based on a mapping of
|
||||
which objects are used by which other objects. Changing this
|
||||
mapping could cause an object to appear in the wrong section, to
|
||||
be written even though it is unreferenced, or to be entirely
|
||||
omitted since, during linearization, we don't enqueue new objects
|
||||
as we traverse for writing.
|
||||
|
||||
* There are several places in QPDFWriter that query a stream's
|
||||
dictionary in order to prepare for writing or to make decisions
|
||||
about certain aspects of the writing process. If the stream data
|
||||
provider has the chance to modify the dictionary, every piece of
|
||||
code that gets stream data would have to be aware of this. This
|
||||
would potentially include end user code. For example, any code
|
||||
that called getDict() on a stream before installing a stream data
|
||||
provider and expected that dictionary to be valid would
|
||||
potentially be broken. As implemented right now, you must perform
|
||||
any modifications on the dictionary in advance and provided
|
||||
/Filter and /DecodeParms at the time you installed the stream
|
||||
data provider. This means that some computations would have to be
|
||||
done more than once, but for linearized files, stream data
|
||||
providers are already called more than once. If the work done by
|
||||
a stream data provider is especially expensive, it can implement
|
||||
its own cache.
|
||||
|
||||
The example examples/pdf-custom-filter.cc demonstrates the use of
|
||||
custom stream filters. This includes a custom pipeline, a custom
|
||||
stream filter, as well as modification of a stream's dictionary to
|
||||
include creation of a new stream that is referenced from
|
||||
/DecodeParms.
|
||||
|
||||
* Removal of raw QPDF* from the API. Discussions in #747 and #754.
|
||||
This is a summary of the arguments I put forth in #754. The idea was
|
||||
to make QPDF::QPDF() private and require all QPDF objects to be
|
||||
shared pointers created with QPDF::create(). This would enable us to
|
||||
have QPDFObjectHandle::getOwningQPDF() return a std::weak_ptr<QPDF>.
|
||||
Prior to #726 (QPDFObject/QPDFValue split, released in qpdf 11.0.0),
|
||||
getOwningQPDF() could return an invalid pointer if the owning QPDF
|
||||
disappeared, but this is no longer the case, which removes the main
|
||||
motivation. QPDF 11 added QPDF::create() anyway though.
|
||||
|
||||
Removing raw QPDF* would look something like this. Note that you
|
||||
can't use std::make_shared<T> unless T has a public constructor.
|
||||
|
||||
QPDF_POINTER_TRANSITION = 0 -- no warnings around calling the QPDF constructor
|
||||
QPDF_POINTER_TRANSITION = 1 -- calls to QPDF() are deprecated, but QPDF is still available so code can be backward compatible and use std::make_shared<QPDF>
|
||||
QPDF_POINTER_TRANSITION = 2 -- the QPDF constructor is private; all calls to std::make_shared<QPDF> have to be replaced with QPDF::create
|
||||
|
||||
If we were to do this, we'd have to look at each use of QPDF* in the
|
||||
interface and decide whether to use a std::shared_ptr or a
|
||||
std::weak_ptr. The answer would almost always be to use a
|
||||
std::weak_ptr, which means we'd have to take the extra step of
|
||||
calling lock(), and it means there would be lots of code changes
|
||||
cause people would have to pass weak pointers instead of raw
|
||||
pointers around, and those have to be constructed and locked.
|
||||
Passing std::shared_ptr around leaves the possibility of creating
|
||||
circular references. It seems to be too much trouble in the library
|
||||
and too much toil for library users to be worth the small benefit of
|
||||
not having to call resetObjGen in QPDF's destructor.
|
||||
|
||||
* Fix Multiple Direct Object Parent Issue
|
||||
|
||||
This idea was rejected because it would be complicated to implement
|
||||
and would likely have a high performance cost to fix what is not
|
||||
really that big of a problem in practice.
|
||||
|
||||
It is possible for a QPDFObjectHandle for a direct object to be
|
||||
contained inside of multiple QPDFObjectHandle objects or even
|
||||
replicated across multiple QPDF objects. This creates a potentially
|
||||
confusing and unintentional aliasing of direct objects. There are
|
||||
known cases in the qpdf library where this happens including page
|
||||
splitting and merging (particularly with page labels, and possibly
|
||||
with other cases), and also with unsafeShallowCopy. Disallowing this
|
||||
would incur a significant performance penalty and is probably not
|
||||
worth doing. If we were to do it, here are some ideas.
|
||||
|
||||
* Add std::weak_ptr<QPDFObject> parent to QPDFObject. When adding a
|
||||
direct object to an array or dictionary, set its parent. When
|
||||
removing it, clear the parent pointer. The parent pointer would
|
||||
always be null for indirect objects, so the parent pointer, which
|
||||
would reside in QPDFObject, would have to be managed by
|
||||
QPDFObjectHandle. This is because QPDFObject can't tell the
|
||||
difference between a resolved indirect object and a direct object.
|
||||
|
||||
* Phase 1: When a direct object that already has a parent is added
|
||||
to a dictionary or array, issue a warning. There would need to be
|
||||
unsafe add methods used by unsafeShallowCopy. These would add but
|
||||
not modify the parent pointer.
|
||||
|
||||
* Phase 2: In the next major release, make the multiple parent case
|
||||
an error. Require people to create a copy. The unsafe operations
|
||||
would still have to be permitted.
|
||||
|
||||
This approach would allow an object to be moved from one object to
|
||||
another by removing it, which returns the now orphaned object, and
|
||||
then inserting it somewhere else. It also doesn't break the pattern
|
||||
of adding a direct object to something and subsequently mutating it.
|
||||
It just prevents the same object from being added to more than one
|
||||
thing.
|
699
TODO.md
Normal file
699
TODO.md
Normal file
|
@ -0,0 +1,699 @@
|
|||
Contents
|
||||
========
|
||||
|
||||
- [Always](#always)
|
||||
- [Next](#always)
|
||||
- [Possible future JSON enhancements](#possible-future-json-enhancements)
|
||||
- [QPDFJob](#qpdfjob)
|
||||
- [Documentation](#documentation)
|
||||
- [Document-level work](#document-level-work)
|
||||
- [Text Appearance Streams](#text-appearance-streams)
|
||||
- [Fuzz Errors](#fuzz-errors)
|
||||
- [External Libraries](#external-libraries)
|
||||
- [ABI Changes](#abi-changes)
|
||||
- [C++ Version Changes](#c-version-changes)
|
||||
- [Page splitting/merging](#page-splittingmerging)
|
||||
- [Analytics](#analytics)
|
||||
- [General](#general)
|
||||
|
||||
- [HISTORICAL NOTES](#historical-notes)
|
||||
|
||||
Always
|
||||
======
|
||||
|
||||
* Evaluate issues tagged with `next` and `bug`. Remember to check discussions and pull requests in
|
||||
addition to regular issues.
|
||||
* When close to release, make sure external-libs is building and follow instructions in
|
||||
../external-libs/README
|
||||
|
||||
Next
|
||||
====
|
||||
|
||||
* Fix #874 -- make args in --encrypt to match the json and make positional fill in the gaps
|
||||
* Maybe fix #553 -- use file times for attachments
|
||||
* std::string_view transition -- work being done by m-holger
|
||||
* Break ground on "Document-level work" -- TODO-pages.md lives on a separate branch.
|
||||
* Standard for CLI and Job JSON support for JSON-based command-line arguments. Come up with a
|
||||
standard way of supporting command-line arguments that take JSON specifications of things so that
|
||||
* there is a predictable way to indicate whether an argument is a file or a JSON blob
|
||||
* with QPDFJob JSON, make sure it is possible to directly include the JSON rather than having to
|
||||
stringify a JSON blob
|
||||
* One option might be to prepend file:// to a filename or otherwise to take a JSON blob. We could
|
||||
have that as a particular type of argument that would behave properly for both job JSON and CLI.
|
||||
|
||||
Possible future JSON enhancements
|
||||
=================================
|
||||
|
||||
* Consider not including unreferenced objects and trimming the trailer in the same way that
|
||||
QPDFWriter does (except don't remove `/ID`). This means excluding the linearization dictionary and
|
||||
hint stream, the encryption dictionary, all keys from trailer that are removed by
|
||||
QPDFWriter::getTrimmedTrailer except `/ID`, any object streams, and the xref stream as long as all
|
||||
those objects are unreferenced. (They always should be, but there could be some bizarre case of
|
||||
someone creating a PDF file that has an indirect reference to one of those, in which case we need
|
||||
to preserve it.) If this is done, make `--preserve-unreferenced` preserve unreference objects and
|
||||
also those extra keys. Search for "linear" and "trailer" in json.rst to update the various places
|
||||
in the documentation that discuss this. Also update the help for --json and
|
||||
--preserve-unreferenced.
|
||||
|
||||
* Add to JSON output the information available from a few additional informational options:
|
||||
|
||||
* --check: add but maybe not by default?
|
||||
|
||||
* --show-linearization: add but maybe not by default? Also figure out whether warnings reported
|
||||
for some of the PDF specs (1.7) are qpdf problems. This may not be worth adding in the first
|
||||
increment.
|
||||
|
||||
* --show-xref: add
|
||||
|
||||
* Consider having --check, --show-encryption, etc., just select the right keys when in json mode. I
|
||||
don't think I want check on by default, so that might be different.
|
||||
|
||||
* Consider having warnings be included in the json in a "warnings" key in json mode.
|
||||
|
||||
QPDFJob
|
||||
=======
|
||||
|
||||
Here are some ideas for QPDFJob that didn't make it into 10.6. Not all of these are necessarily
|
||||
good -- just things to consider.
|
||||
|
||||
* How do we chain jobs? The idea would be that the input and/or output of a QPDFJob could be a QPDF
|
||||
object rather than a file. For input, it's pretty easy. For output, none of the output-specific
|
||||
options (encrypt, compress-streams, objects-streams, etc.) would have any affect, so we would have
|
||||
to treat this like inspect for error checking. The QPDF object in the state where it's ready to be
|
||||
sent off to QPDFWriter would be used as the input to the next QPDFJob. For the job json, I think
|
||||
we can have the output be an identifier that can be used as the input for another QPDFJob. For a
|
||||
json file, we could the top level detect if it's an array with the convention that exactly one has
|
||||
an output, or we could have a subkey with other job definitions or something. Ideally, any input
|
||||
(copy-attachments-from, pages, etc.) could use a QPDF object. It wouldn't surprise me if this
|
||||
exposes bugs in qpdf around foreign streams as this has been a relatively fragile area before.
|
||||
|
||||
Documentation
|
||||
=============
|
||||
|
||||
* Do a full pass through the documentation.
|
||||
|
||||
* Make sure `qpdf` is consistent. Use QPDF when just referring to the package.
|
||||
* Make sure markup is consistent
|
||||
* Autogenerate where possible
|
||||
* Consider which parts might be good candidates for moving to the wiki.
|
||||
|
||||
* Commit 'Manual - enable line wrapping in table cells' from Mon Jan 17 12:22:35 2022 +0000 enables
|
||||
table cell wrapping. See if this can be incorporated directly into sphinx_rtd_theme and the
|
||||
workaround can be removed.
|
||||
|
||||
* When possible, update the debian package to include docs again. See
|
||||
https://bugs.debian.org/1004159 for details.
|
||||
|
||||
Document-level work
|
||||
===================
|
||||
|
||||
* Ideas here may by superseded by #593.
|
||||
|
||||
* QPDFPageCopier -- object for moving pages around within files or between files and performing
|
||||
various transformations. Reread/rewrite
|
||||
_page-selection in the manual if needed.
|
||||
|
||||
* Handle all the stuff of pages and split-pages
|
||||
* Do n-up, booklet, collation
|
||||
* Look through cli and see what else...flatten-*?
|
||||
* See comments in QPDFPageDocumentHelper.hh for addPage -- search for "a future version".
|
||||
* Make it efficient for bulk operations
|
||||
* Make certain doc-level features selectable
|
||||
* qpdf.cc should do all its page operations, including overlay/underlay, splitting, and merging,
|
||||
using this
|
||||
* There should also be example code
|
||||
|
||||
* After doc-level checks are in, call --check on the output files in the "Copy Annotations" tests.
|
||||
|
||||
* Document-level checks. For example, for forms, make sure all form fields point to an annotation on
|
||||
exactly one page as well as that all widget annotations are associated with a form field. Hook
|
||||
this into QPDFPageCopier as well as the doc helpers. Make sure it is called from --check.
|
||||
|
||||
* See also issues tagged with "pages". Include closed issues.
|
||||
|
||||
* Add flags to CLI to select which document-level options to preserve or not preserve. We will
|
||||
probably need a pair of mutually exclusive, repeatable options with a way to specify all, none,
|
||||
only {x,y}, or all but {x,y}.
|
||||
|
||||
* If a page contains a reference a file attachment annotation, when that page is copied, if the file
|
||||
attachment appears in the top-level EmbeddedFiles tree, that entry should be preserved in the
|
||||
destination file. Otherwise, we probably will require the use of --copy-attachments-from to
|
||||
preserve these. What will the strategy be for deduplicating in the automatic case?
|
||||
|
||||
Text Appearance Streams
|
||||
=======================
|
||||
|
||||
This is a list of known issues with text appearance streams and things we might do about it.
|
||||
|
||||
* For variable text, the spec says to pull any resources from /DR that are referenced in /DA but if
|
||||
the resource dictionary already has that resource, just use the one that's there. The current code
|
||||
looks only for /Tf and adds it if needed. We might want to instead merge /DR with resources and
|
||||
then remove anything that's unreferenced. We have all the code required for that in ResourceFinder
|
||||
except TfFinder also gets the font size, which ResourceFinder doesn't do.
|
||||
|
||||
* There are things we are missing because we don't look at font metrics. The code from TextBuilder
|
||||
(work) has almost everything in it that is required. Once we have knowledge of character widths,
|
||||
we can support quadding and multiline text fields (/Ff 4096), and we can potentially squeeze text
|
||||
to fit into a field. For multiline, first squeeze vertically down to the font height, then squeeze
|
||||
horizontally with Tz. For single line, squeeze horizontally with Tz. If we use Tz, issue a
|
||||
warning.
|
||||
|
||||
* When mapping characters to widths, we will need to care about character encoding. For built-in
|
||||
fonts, we can create a map from Unicode code point to width and then go from the font's encoding
|
||||
to unicode to the width. See misc/character-encoding/ (not on github)
|
||||
and font metric information for the 14 standard fonts in my local pdf-spec directory.
|
||||
|
||||
* Once we know about character widths, we can correctly support auto-sized variable text fields
|
||||
(0 Tf). If this is fixed, search for "auto-sized" in cli.rst.
|
||||
|
||||
Fuzz Errors
|
||||
===========
|
||||
|
||||
* https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=<N>
|
||||
|
||||
* Ignoring these:
|
||||
* Out of memory in dct: 35001, 32516
|
||||
|
||||
External Libraries
|
||||
==================
|
||||
|
||||
Current state (10.0.2):
|
||||
|
||||
* qpdf/external-libs repository builds external-libs on a schedule. It detects and downloads the
|
||||
latest versions of zlib, jpeg, and openssl and creates source and binary distribution zip files in
|
||||
an artifact called "distribution".
|
||||
|
||||
* Releases in qpdf/external-libs are made manually. They contain qpdf-external-libs-{bin,src}.zip.
|
||||
|
||||
* The qpdf build finds the latest non-prerelease release and downloads the qpdf-external-libs-*.zip
|
||||
files from the releases in the setup stage.
|
||||
|
||||
* To upgrade to a new version of external-libs, create a new release of qpdf/external-libs (see
|
||||
README-maintainer in external-libs) from the distribution artifact of the most recent successful
|
||||
build after ensuring that it works.
|
||||
|
||||
Desired state:
|
||||
|
||||
* The qpdf/external-libs repository should create release candidates. Ideally, every scheduled run
|
||||
would make its zip files available. A personal access token with actions:read scope for the
|
||||
qpdf/external-libs repository is required to download the artifact from an action run, and
|
||||
qpdf/qpdf's secrets.GITHUB_TOKEN doesn't have this access. We could create a service account for
|
||||
this purpose. As an alternative, we could have a draft release in qpdf/external-libs that the
|
||||
qpdf/external-libs build could update with each candidate. It may also be possible to solve this
|
||||
by developing a simple GitHub app.
|
||||
|
||||
* Scheduled runs of the qpdf build in the qpdf/qpdf repository (not a fork or pull request) could
|
||||
download external-libs from the release candidate area instead of the latest stable release.
|
||||
Pushes to the build branch should still use the latest release so it always matches the main
|
||||
branch.
|
||||
|
||||
* Periodically, we would create a release of external-libs from the release candidate zip files.
|
||||
This could be done safely because we know the latest qpdf works with it. This could be done at
|
||||
least before every release of qpdf, but potentially it could be done at other times, such as when
|
||||
a new dependency version is available or after some period of time.
|
||||
|
||||
Other notes:
|
||||
|
||||
* The external-libs branch in qpdf/qpdf was never documented. We might be able to get away with
|
||||
deleting it.
|
||||
|
||||
* See README-maintainer in qpdf/external-libs for information on creating a release. This could be
|
||||
at least partially scripted in a way that works for the qpdf/qpdf repository as well since they
|
||||
are very similar.
|
||||
|
||||
ABI Changes
|
||||
===========
|
||||
|
||||
This is a list of changes to make next time there is an ABI change. Comments appear in the code
|
||||
prefixed by "ABI".
|
||||
|
||||
Always:
|
||||
* Search for ABI in source and header files
|
||||
* Search for "[[deprecated" to find deprecated APIs that can be removed
|
||||
* Search for issues, pull requests, and discussions with the "abi" label
|
||||
* Check discussion "qpdf X planning" where X is the next major version. This should be tagged `abi`
|
||||
|
||||
For qpdf 12, see https://github.com/qpdf/qpdf/discussions/785
|
||||
|
||||
C++ Version Changes
|
||||
===================
|
||||
|
||||
Use
|
||||
```
|
||||
// C++NN: ...
|
||||
```
|
||||
to mark places in the code that should be updated when we require at least that version of C++.
|
||||
|
||||
Page splitting/merging
|
||||
======================
|
||||
|
||||
* Update page splitting and merging to handle document-level constructs with page impact such as
|
||||
interactive forms and article threading. Check keys in the document catalog for others, such as
|
||||
outlines, page labels, thumbnails, and zones. For threads, Subramanyam provided a test file; see
|
||||
../misc/article-threads.pdf. Email Q-Count: 431864 from 2009-11-03.
|
||||
|
||||
* bookmarks (outlines) 12.3.3
|
||||
* support bookmarks when merging
|
||||
* prune bookmarks that don't point to a surviving page when merging or splitting
|
||||
* make sure conflicting named destinations work possibly test by including the same file by two
|
||||
paths in a merge
|
||||
* see also comments in issue 343
|
||||
|
||||
Note: original implementation of bookmark preservation for split pages caused a very high
|
||||
performance hit. The problem was introduced in 313ba081265f69ac9a0324f9fe87087c72918191 and
|
||||
reverted in the commit that adds this paragraph. The revert includes marking a few tests cases as
|
||||
$td->EXPECT_FAILURE. When properly coded, the test cases will need to be adjusted to only include
|
||||
the parts of the outlines that are actually copied. The tests in question are
|
||||
"split page with outlines". When implementing properly, ensure that the performance is not
|
||||
adversely affected by timing split-pages on a large file with complex outlines such as the PDF
|
||||
specification.
|
||||
|
||||
When pruning outlines, keep all outlines in the hierarchy that are above an outline for a page we
|
||||
care about. If one of the ancestor outlines points to a non-existent page, clear its dest. If an
|
||||
outline does not have any children that point to pages in the document, just omit it.
|
||||
|
||||
Possible strategy:
|
||||
* resolve all named destinations to explicit destinations
|
||||
* concatenate top-level outlines
|
||||
* prune outlines whose dests don't point to a valid page
|
||||
* recompute all /Count fields
|
||||
|
||||
Test files
|
||||
* page-labels-and-outlines.pdf: old file with both page labels and outlines. All destinations are
|
||||
explicit destinations. Each page has Potato and a number. All titles are feline names.
|
||||
* outlines-with-actions.pdf: mixture of explicit destinations, named destinations, goto actions
|
||||
with explicit destinations, and goto actions with named destinations; uses /Dests key in names
|
||||
dictionary. Each page has Salad and a number. All titles are silly words. One destination is an
|
||||
indirect object.
|
||||
* outlines-with-old-root-dests.pdf: like outlines-with-actions except it uses the PDF-1.1 /Dests
|
||||
dictionary for named destinations, and each page has Soup and a number. Also pages are numbered
|
||||
with upper-case Roman numerals starting with 0. All titles are silly words preceded by a bullet.
|
||||
|
||||
If outline handling is significantly improved, see ../misc/bad-outlines/bad-outlines.pdf and
|
||||
email:
|
||||
https://mail.google.com/mail/u/0/#search/rfc822msgid%3A02aa01d3d013%249f766990%24de633cb0%24%40mono.hr)
|
||||
|
||||
* Form fields: should be similar to outlines.
|
||||
|
||||
Analytics
|
||||
=========
|
||||
|
||||
Consider features that make it easier to detect certain patterns in PDF files. The information below
|
||||
could be computed using an external program that reads the existing json, but if it's useful enough,
|
||||
we could add it directly to the json output.
|
||||
|
||||
* Add to "pages" in the json:
|
||||
* "inheritsresources": bool; whether there are any inherited attributes from ancestor page tree
|
||||
nodes
|
||||
* "sharedresources": a list of indirect objects that are
|
||||
"/Resources" dictionaries or "XObject" resource dictionary subkeys of either the page itself or
|
||||
of any form XObject referenced by the page.
|
||||
|
||||
* Add to "objectinfo" in json: "directpagerefcount": the number of pages that directly reference
|
||||
this object (i.e., you can find an indirect reference to the object in the page dictionary without
|
||||
traversing over any indirect objects)
|
||||
|
||||
General
|
||||
=======
|
||||
|
||||
NOTE: Some items in this list refer to files in my personal home directory or that are otherwise not
|
||||
publicly accessible. This includes things sent to me by email that are specifically not public. Even
|
||||
so, I find it useful to make reference to them in this list.
|
||||
|
||||
* Consider enabling code scanning on GitHub.
|
||||
|
||||
* Add an option --ignore-encryption to ignore encryption information and treat encrypted files as if
|
||||
they weren't encrypted. This should make it possible to solve #598 (--show-encryption without a
|
||||
password). We'll need to make sure we don't try to filter any streams in this mode. Ideally we
|
||||
should be able to combine this with --json so we can look at the raw encrypted strings and streams
|
||||
if we want to, though be sure to document that the resulting JSON won't be convertible back to a
|
||||
valid PDF. Since providing the password may reveal additional details, --show-encryption could
|
||||
potentially retry with this option if the first time doesn't work. Then, with the file open, we
|
||||
can read the encryption dictionary normally. If this is done, search for "raw, encrypted" in
|
||||
json.rst.
|
||||
|
||||
* In libtests, separate executables that need the object library from those that strictly use public
|
||||
API. Move as many of the test drivers from the qpdf directory into the latter category as long as
|
||||
doing so isn't too troublesome from a coverage standpoint.
|
||||
|
||||
* Consider generating a non-flat pages tree before creating output to better handle files with lots
|
||||
of pages. If there are more than 256 pages, add a second layer with the second layer nodes having
|
||||
no more than 256 nodes and being as evenly sizes as possible. Don't worry about the case of more
|
||||
than 65,536 pages. If the top node has more than 256 children, we'll live with it. This is only
|
||||
safe if all intermediate page nodes have only /Kids, /Parent, /Type, and /Count.
|
||||
|
||||
* Look at https://bestpractices.coreinfrastructure.org/en
|
||||
|
||||
* Consider adding fuzzer code for JSON
|
||||
|
||||
* Rework tests so that nothing is written into the source directory. Ideally then the entire build
|
||||
could be done with a read-only source tree.
|
||||
|
||||
* Large file tests fail with linux32 before and after cmake. This was first noticed after 10.6.3. I
|
||||
don't think it's worth fixing.
|
||||
|
||||
* Consider updating the fuzzer with code that exercises copyAnnotations, file attachments, and name
|
||||
and number trees. Check fuzzer coverage.
|
||||
|
||||
* Add code for creation of a file attachment annotation. It should also be possible to create a
|
||||
widget annotation and a form field. Update the pdf-attach-file.cc example with new APIs when
|
||||
ready.
|
||||
|
||||
* Flattening of form XObjects seems like something that would be useful in the library. We are
|
||||
seeing more cases of completely valid PDF files with form XObjects that cause problems in other
|
||||
software. Flattening of form XObjects could be a useful way to work around those issues or to
|
||||
prepare files for additional processing, making it possible for users of the qpdf library to not
|
||||
be concerned about form XObjects. This could be done recursively; i.e., we could have a method to
|
||||
embed a form XObject into whatever contains it, whether that is a form XObject or a page. This
|
||||
would require more significant interpretation of the content stream. We would need a test file in
|
||||
which the placement of the form XObject has to be in the right place, e.g., the form XObject
|
||||
partially obscures earlier code and is partially obscured by later code. Keys in the resource
|
||||
dictionary may need to be changed -- create test cases with lots of duplicated/overlapping keys.
|
||||
|
||||
* Part of closed_file_input_source.cc is disabled on Windows because of odd failures. It might be
|
||||
worth investigating so we can fully exercise this in the test suite. That said,
|
||||
ClosedFileInputSource is exercised elsewhere in qpdf's test suite, so this is not that pressing.
|
||||
|
||||
* If possible, consider adding CCITT3, CCITT4, or any other easy filters. For some reference code
|
||||
that we probably can't use but may be handy anyway, see
|
||||
http://partners.adobe.com/public/developer/ps/sdk/index_archive.html
|
||||
|
||||
* If possible, support the following types of broken files:
|
||||
|
||||
- Files that have no whitespace token after "endobj" such that endobj collides with the start of
|
||||
the next object
|
||||
|
||||
- See ../misc/broken-files
|
||||
|
||||
- See ../misc/bad-files-issue-476. This directory contains a snapshot of the google doc and linked
|
||||
PDF files from issue #476. Please see the issue for details.
|
||||
|
||||
* Additional form features
|
||||
* set value from CLI? Specify title, and provide way to disambiguate, probably by giving objgen of
|
||||
field
|
||||
|
||||
* Pl_TIFFPredictor is pretty slow.
|
||||
|
||||
* Support for handling file names with Unicode characters in Windows is incomplete. qpdf seems to
|
||||
support them okay from a functionality standpoint, and the right thing happens if you pass in
|
||||
UTF-8 encoded filenames to QPDF library routines in Windows (they are converted internally to
|
||||
wchar_t*), but file names are encoded in UTF-8 on output, which doesn't produce nice error
|
||||
messages or output on Windows in some cases.
|
||||
|
||||
* If we ever wanted to do anything more with character encoding, see ../misc/character-encoding/,
|
||||
which includes machine-readable dump of table D.2 in the ISO-32000 PDF spec. This shows the
|
||||
mapping between Unicode, StandardEncoding, WinAnsiEncoding, MacRomanEncoding, and PDFDocEncoding.
|
||||
|
||||
* Some test cases on bad files fail because qpdf is unable to find the root dictionary when it fails
|
||||
to read the trailer. Recovery could find the root dictionary and even the info dictionary in other
|
||||
ways. In particular, issue-202.pdf can be opened by evince, and there's no real reason that qpdf
|
||||
couldn't be made to be able to recover that file as well.
|
||||
|
||||
* Audit every place where qpdf allocates memory to see whether there are cases where malicious
|
||||
inputs could cause qpdf to attempt to grab very large amounts of memory. Certainly there are cases
|
||||
like this, such as if a very highly compressed, very large image stream is requested in a buffer.
|
||||
Hopefully normal input to output filtering doesn't ever try to do this. QPDFWriter should be
|
||||
checked carefully too. See also bugs/private/from-email-663916/
|
||||
|
||||
* Interactive form modification:
|
||||
https://github.com/qpdf/qpdf/issues/213 contains a good discussion of some ideas for adding
|
||||
methods to modify annotations and form fields if we want to make it easier to support
|
||||
modifications to interactive forms. Some of the ideas have been implemented, and some of the
|
||||
probably never will be implemented, but it's worth a read if there is an intention to work on
|
||||
this. In the issue, search for "Regarding write functionality", and read that comment and the
|
||||
responses to it.
|
||||
|
||||
* Look at ~/Q/pdf-collection/forms-from-appian/
|
||||
|
||||
* When decrypting files with /R=6, hash_V5 is called more than once with the same inputs. Caching
|
||||
the results or refactoring to reduce the number of identical calls could improve performance for
|
||||
workloads that involve processing large numbers of small files.
|
||||
|
||||
* Consider adding a method to balance the pages tree. It would call pushInheritedAttributesToPage,
|
||||
construct a pages tree from scratch, and replace the /Pages key of the root dictionary with the
|
||||
new tree.
|
||||
|
||||
* Study what's required to support savable forms that can be saved by Adobe Reader. Does this
|
||||
require actually signing the document with an Adobe private key? Search for "Digital signatures"
|
||||
in the PDF spec, and look at ~/Q/pdf-collection/form-with-full-save.pdf, which came from Adobe's
|
||||
example site. See also ../misc/digital-sign-from-trueroad/ and
|
||||
../misc/digital-signatures/digitally-signed-pdf-xfa.pdf. If digital signatures are implemented,
|
||||
update the docs on crypto providers, which mention that this may happen in the future.
|
||||
|
||||
* Qpdf does not honor /EFF when adding new file attachments. When it encrypts, it never generates
|
||||
streams with explicit crypt filters. Prior to 10.2, there was an incorrect attempt to treat /EFF
|
||||
as a default value for decrypting file attachment streams, but it is not supposed to mean that.
|
||||
Instead, it is intended for conforming writers to obey this when adding new attachments. Qpdf is
|
||||
not a conforming writer in that respect.
|
||||
|
||||
* The whole xref handling code in the QPDF object allows the same object with more than one
|
||||
generation to coexist, but a lot of logic assumes this isn't the case. Anything that creates
|
||||
mappings only with the object number and not the generation is this way, including most of the
|
||||
interaction between QPDFWriter and QPDF. If we wanted to allow the same object with more than one
|
||||
generation to coexist, which I'm not sure is allowed, we could fix this by changing xref_table.
|
||||
Alternatively, we could detect and disallow that case. In fact, it appears that Adobe reader and
|
||||
other PDF viewing software silently ignores objects of this type, so this is probably not a big
|
||||
deal.
|
||||
|
||||
* From a suggestion in bug 3152169, consider having an option to re-encode inline images with an
|
||||
ASCII encoding.
|
||||
|
||||
* From github issue 2, provide more in-depth output for examining hint stream contents. Consider
|
||||
adding on option to provide a human-readable dump of linearization hint tables. This should
|
||||
include improving the 'overflow reading bit stream' message as reported in issue #2. There are
|
||||
multiple calls to stopOnError in the linearization checking code. Ideally, these should not
|
||||
terminate checking. It would require re-acquiring an understanding of all that code to make the
|
||||
checks more robust. In particular, it's hard to look at the code and quickly determine what is a
|
||||
true logic error and what could happen because of malformed user input. See also
|
||||
../misc/linearization-errors.
|
||||
|
||||
* If I ever decide to make appearance stream-generation aware of fonts or font metrics, see email
|
||||
from Tobias with Message-ID
|
||||
<5C3C9C6C.8000102@thax.hardliners.org> dated 2019-01-14.
|
||||
|
||||
* Look at places in the code where object traversal is being done and, where possible, try to avoid
|
||||
it entirely or at least avoid ever traversing the same objects multiple times.
|
||||
|
||||
----------------------------------------------------------------------
|
||||
|
||||
### HISTORICAL NOTES
|
||||
|
||||
* [Performance](#performance)
|
||||
* [QPDFPagesTree](#qpdfpagestree)
|
||||
* [Rejected Ideas](#rejected-ideas)
|
||||
|
||||
Performance
|
||||
===========
|
||||
|
||||
As described in https://github.com/qpdf/qpdf/issues/401, there was great performance degradation
|
||||
between qpdf 7.1.1 and 9.1.1. Doing a bisect between dac65a21fb4fa5f871e31c314280b75adde89a6c and
|
||||
release-qpdf-7.1.1, I found several commits that damaged performance. I fixed some of them to
|
||||
improve performance by about 70% (as measured by saying that old times were 170% of new times). The
|
||||
remaining commits that broke performance either can't be correct because they would re-introduce an
|
||||
old bug or aren't worth correcting because of the high value they offer relative to a relatively low
|
||||
penalty. For historical reference, here are the commits. The numbers are the time in seconds on the
|
||||
machine I happened to be using of splitting the first 100 pages of PDF32000_2008.pdf 20 times and
|
||||
taking an average duration.
|
||||
|
||||
Commits that broke performance:
|
||||
|
||||
* d0e99f195a987c483bbb6c5449cf39bee34e08a1 -- object description and context: 0.39 -> 0.45
|
||||
* a01359189b32c60c2d55b039f7aefd6c3ce0ebde (minus 313ba08) -- fix dangling references: 0.55 -> 0.6
|
||||
* e5f504b6c5dc34337cc0b316b4a7b1fca7e614b1 -- sparse array: 0.6 -> 0.62
|
||||
|
||||
Other intermediate steps that were previously fixed:
|
||||
|
||||
* 313ba081265f69ac9a0324f9fe87087c72918191 -- copy outlines into split: 0.55 -> 4.0
|
||||
* a01359189b32c60c2d55b039f7aefd6c3ce0ebde -- fix dangling references:
|
||||
4.0 -> 9.0
|
||||
|
||||
This commit fixed the awful problem introduced in 313ba081:
|
||||
|
||||
* a5a016cdd26a8e5c99e5f019bc30d1bdf6c050a2 -- revert outline preservation: 9.0 -> 0.6
|
||||
|
||||
Note that the fix dangling references commit had a much worse impact prior to removing the outline
|
||||
preservation, so I also measured its impact in isolation.
|
||||
|
||||
A few important lessons (in README-maintainer)
|
||||
|
||||
* Indirection through PointerHolder<Members> is expensive, and should not be used for things that
|
||||
are created and destroyed frequently such as QPDFObjectHandle and QPDFObject.
|
||||
* Traversal of objects is expensive and should be avoided where possible.
|
||||
|
||||
Also, it turns out that PointerHolder is more performant than std::shared_ptr. (This was true at the
|
||||
time but subsequent implementations of std::shared_ptr became much more efficient.)
|
||||
|
||||
QPDFPagesTree
|
||||
=============
|
||||
|
||||
On a few occasions, I have considered implementing a QPDFPagesTree object that would allow the
|
||||
document's original page tree structure to be preserved. See comments at the top QPDF_pages.cc for
|
||||
why this was abandoned.
|
||||
|
||||
Partial work is in refs/attic/QPDFPagesTree. QPDFPageTree is mostly implemented and mostly tested.
|
||||
There are not enough cases of different kinds of operations (pclm, linearize, json, etc.) with
|
||||
non-flat pages trees. Insertion is not implemented. Insertion is potentially complex because of the
|
||||
issue of inherited objects. We will have to call pushInheritedAttributesToPage before adding any
|
||||
pages to the pages tree. The test suite is failing on that branch.
|
||||
|
||||
Some parts of page tree repair are silent (no warnings). All page tree repair should warn. The
|
||||
reason is that page tree repair will change object numbers, and knowing that is important when
|
||||
working with JSON output.
|
||||
|
||||
If we were to do this, we would still need keep a pages cache for efficient insertion. There's no
|
||||
reason we can't keep a vector of page objects up to date and just do a traversal the first time we
|
||||
do getAllPages just like we do now. The difference is that we would not flatten the pages tree. It
|
||||
would be useful to go through QPDF_pages and reimplement everything without calling
|
||||
flattenPagesTree. Then we can remove flattenPagesTree, which is private. That said, with the
|
||||
addition of creating non-flat pages trees, there is really no reason not to flatten the pages tree
|
||||
for internal use.
|
||||
|
||||
In its current state, QPDFPagesTree does not proactively fix /Type or correct page objects that are
|
||||
used multiple times. You have to traverse the pages tree to trigger this operation. It would be nice
|
||||
if we would do that somewhere but not do it more often than necessary so isPagesObject and
|
||||
isPageObject are reliable and can be made more reliable. Maybe add a validate or repair function? It
|
||||
should also make sure /Count and /Parent are correct.
|
||||
|
||||
Rejected Ideas
|
||||
==============
|
||||
|
||||
* Investigate whether there is a way to automate the memory checker tests for Windows.
|
||||
|
||||
* Provide support in QPDFWriter for writing incremental updates. Provide support in qpdf for
|
||||
preserving incremental updates. The goal should be that QDF mode should be fully functional for
|
||||
files with incremental updates including fix_qdf.
|
||||
|
||||
Note that there's nothing that says an indirect object in one update can't refer to an object that
|
||||
doesn't appear until a later update. This means that QPDF has to treat indirect null objects
|
||||
differently from how it does now. QPDF drops indirect null objects that appear as members of
|
||||
arrays or dictionaries. For arrays, it's handled in QPDFWriter where we make indirect nulls
|
||||
direct. This is in a single if block, and nothing else in the code cares about it. We could just
|
||||
remove that if block and not break anything except a few test cases that exercise the current
|
||||
behavior. For dictionaries, it's more complicated. In this case, QPDF_Dictionary::getKeys()
|
||||
ignores all keys with null values, and hasKey() returns false for keys that have null values. We
|
||||
would probably want to make QPDF_Dictionary able to handle the special case of keys that are
|
||||
indirect nulls and basically never have it drop any keys that are indirect objects.
|
||||
|
||||
If we make a change to have qpdf preserve indirect references to null objects, we have to note
|
||||
this in ChangeLog and in the release notes since this will change output files. We did this before
|
||||
when we stopped flattening scalar references, so this is probably not a big deal. We also have to
|
||||
make sure that the testing for this handles non-trivial cases of the targets of indirect nulls
|
||||
being replaced by real objects in an update. I'm not sure how this plays with linearization, if at
|
||||
all. For cases where incremental updates are not being preserved as incremental updates and where
|
||||
the data is being folded in (as is always the case with qpdf now), none of this should make any
|
||||
difference in the actual semantics of the files.
|
||||
|
||||
* The second xref stream for linearized files has to be padded only because we need file_size as
|
||||
computed in pass 1 to be accurate. If we were not allowing writing to a pipe, we could seek back
|
||||
to the beginning and fill in the value of /L in the linearization dictionary as an optimization to
|
||||
alleviate the need for this padding. Doing so would require us to pad the /L value individually
|
||||
and also to save the file descriptor and determine whether it's seekable. This is probably not
|
||||
worth bothering with.
|
||||
|
||||
* Based on an idea suggested by user "Atom Smasher", consider providing some mechanism to recover
|
||||
earlier versions of a file embedded prior to appended sections.
|
||||
|
||||
* Consider creating a sanitizer to make it easier for people to send broken files. Now that we have
|
||||
json mode, this is probably no longer worth doing. Here is the previous idea, possibly implemented
|
||||
by making it possible to run the lexer (tokenizer) over a whole file. Make it possible to replace
|
||||
all strings in a file lexically even on badly broken files. Ideally this should work files that
|
||||
are lacking xref, have broken links, duplicated dictionary keys, syntax errors, etc., and ideally
|
||||
it should work with encrypted files if possible. This should go through the streams and strings
|
||||
and replace them with fixed or random characters, preferably, but not necessarily, in a manner
|
||||
that works with fonts. One possibility would be to detect whether a string contains characters
|
||||
with normal encoding, and if so, use 0x41. If the string uses character maps, use 0x01. The output
|
||||
should otherwise be unrelated to the input. This could be built after the filtering and tokenizer
|
||||
rewrite and should be done in a manner that takes advantage of the other lexical features. This
|
||||
sanitizer should also clear metadata and replace images. If I ever do this, the file from issue
|
||||
#494 would be a great one to look at.
|
||||
|
||||
* Here are some notes about having stream data providers modify stream dictionaries. I had wanted to
|
||||
add this functionality to make it more efficient to create stream data providers that may
|
||||
dynamically decide what kind of filters to use and that may end up modifying the dictionary
|
||||
conditionally depending on the original stream data. Ultimately I decided not to implement this
|
||||
feature. This paragraph describes why.
|
||||
|
||||
* When writing, the way objects are placed into the queue for writing strongly precludes creation
|
||||
of any new indirect objects, or even changing which indirect objects are referenced from which
|
||||
other objects, because we sometimes write as we are traversing and enqueuing objects. For
|
||||
non-linearized files, there is a risk that an indirect object that used to be referenced would
|
||||
no longer be referenced, and whether it was already written to the output file would be based on
|
||||
an accident of where it was encountered when traversing the object structure. For linearized
|
||||
files, the situation is considerably worse. We decide which section of the file to write an
|
||||
object to based on a mapping of which objects are used by which other objects. Changing this
|
||||
mapping could cause an object to appear in the wrong section, to be written even though it is
|
||||
unreferenced, or to be entirely omitted since, during linearization, we don't enqueue new
|
||||
objects as we traverse for writing.
|
||||
|
||||
* There are several places in QPDFWriter that query a stream's dictionary in order to prepare for
|
||||
writing or to make decisions about certain aspects of the writing process. If the stream data
|
||||
provider has the chance to modify the dictionary, every piece of code that gets stream data
|
||||
would have to be aware of this. This would potentially include end user code. For example, any
|
||||
code that called getDict() on a stream before installing a stream data provider and expected
|
||||
that dictionary to be valid would potentially be broken. As implemented right now, you must
|
||||
perform any modifications on the dictionary in advance and provided /Filter and /DecodeParms at
|
||||
the time you installed the stream data provider. This means that some computations would have to
|
||||
be done more than once, but for linearized files, stream data providers are already called more
|
||||
than once. If the work done by a stream data provider is especially expensive, it can implement
|
||||
its own cache.
|
||||
|
||||
The example examples/pdf-custom-filter.cc demonstrates the use of custom stream filters. This
|
||||
includes a custom pipeline, a custom stream filter, as well as modification of a stream's
|
||||
dictionary to include creation of a new stream that is referenced from /DecodeParms.
|
||||
|
||||
* Removal of raw QPDF* from the API. Discussions in #747 and #754. This is a summary of the
|
||||
arguments I put forth in #754. The idea was to make QPDF::QPDF() private and require all QPDF
|
||||
objects to be shared pointers created with QPDF::create(). This would enable us to have
|
||||
QPDFObjectHandle::getOwningQPDF() return a std::weak_ptr<QPDF>. Prior to #726 (
|
||||
QPDFObject/QPDFValue split, released in qpdf 11.0.0), getOwningQPDF() could return an invalid
|
||||
pointer if the owning QPDF disappeared, but this is no longer the case, which removes the main
|
||||
motivation. QPDF 11 added QPDF::create() anyway though.
|
||||
|
||||
Removing raw QPDF* would look something like this. Note that you can't use std::make_shared<T>
|
||||
unless T has a public constructor.
|
||||
|
||||
QPDF_POINTER_TRANSITION = 0 -- no warnings around calling the QPDF constructor
|
||||
QPDF_POINTER_TRANSITION = 1 -- calls to QPDF() are deprecated, but QPDF is still available so code
|
||||
can be backward compatible and use std::make_shared<QPDF>
|
||||
QPDF_POINTER_TRANSITION = 2 -- the QPDF constructor is private; all calls to
|
||||
std::make_shared<QPDF> have to be replaced with QPDF::create
|
||||
|
||||
If we were to do this, we'd have to look at each use of QPDF* in the interface and decide whether
|
||||
to use a std::shared_ptr or a std::weak_ptr. The answer would almost always be to use a std::
|
||||
weak_ptr, which means we'd have to take the extra step of calling lock(), and it means there would
|
||||
be lots of code changes cause people would have to pass weak pointers instead of raw pointers
|
||||
around, and those have to be constructed and locked. Passing std::shared_ptr around leaves the
|
||||
possibility of creating circular references. It seems to be too much trouble in the library and
|
||||
too much toil for library users to be worth the small benefit of not having to call resetObjGen in
|
||||
QPDF's destructor.
|
||||
|
||||
* Fix Multiple Direct Object Parent Issue
|
||||
|
||||
This idea was rejected because it would be complicated to implement and would likely have a high
|
||||
performance cost to fix what is not really that big of a problem in practice.
|
||||
|
||||
It is possible for a QPDFObjectHandle for a direct object to be contained inside of multiple
|
||||
QPDFObjectHandle objects or even replicated across multiple QPDF objects. This creates a
|
||||
potentially confusing and unintentional aliasing of direct objects. There are known cases in the
|
||||
qpdf library where this happens including page splitting and merging (particularly with page
|
||||
labels, and possibly with other cases), and also with unsafeShallowCopy. Disallowing this would
|
||||
incur a significant performance penalty and is probably not worth doing. If we were to do it, here
|
||||
are some ideas.
|
||||
|
||||
* Add std::weak_ptr<QPDFObject> parent to QPDFObject. When adding a direct object to an array or
|
||||
dictionary, set its parent. When removing it, clear the parent pointer. The parent pointer would
|
||||
always be null for indirect objects, so the parent pointer, which would reside in QPDFObject,
|
||||
would have to be managed by QPDFObjectHandle. This is because QPDFObject can't tell the
|
||||
difference between a resolved indirect object and a direct object.
|
||||
|
||||
* Phase 1: When a direct object that already has a parent is added to a dictionary or array, issue
|
||||
a warning. There would need to be unsafe add methods used by unsafeShallowCopy. These would add
|
||||
but not modify the parent pointer.
|
||||
|
||||
* Phase 2: In the next major release, make the multiple parent case an error. Require people to
|
||||
create a copy. The unsafe operations would still have to be permitted.
|
||||
|
||||
This approach would allow an object to be moved from one object to another by removing it, which
|
||||
returns the now orphaned object, and then inserting it somewhere else. It also doesn't break the
|
||||
pattern of adding a direct object to something and subsequently mutating it. It just prevents the
|
||||
same object from being added to more than one thing.
|
|
@ -137,7 +137,7 @@ for i in appdir/usr/share/doc/qpdf; do
|
|||
cp $top/LICENSE.txt $i
|
||||
cp $top/Artistic-2.0 $i/Artistic-LICENSE.txt
|
||||
cp $top/ChangeLog $i/README-ChangeLog
|
||||
cp $top/TODO $i/README-todo
|
||||
cp $top/TODO.md $i/README-todo.md
|
||||
done
|
||||
|
||||
# The following lines are experimental (for debugging; and to test
|
||||
|
|
Loading…
Reference in New Issue
Block a user