Prepare release notes for 10.6 (so far)

ChangeLog

@@ -68,7 +68,7 @@
 	object.
 
 	* A light C API around basic QPDFJob functionality is in
-	include/qpdf/qpdf-job-c.h.p
+	include/qpdf/qpdfjob-c.h.p
 
 	* Add new functions version of QUtil::call_main_from_wmain that
 	takes a constant argv array.
@@ -79,10 +79,10 @@
 	description to --help and the manual.
 
 	* The --json flag now takes a version number as an optional
-	parameter. The default will remain version 1 for compatibility.
+	parameter. The default will remain version 1 for compatibility
-	This enables future code to use --json=latest to always get the
+	until the release of qpdf 11, after which it will become "latest".
-	latest version or to use a specific version. At this time, there's
+	At this time, there's only version 1, but a version 2 may appear
-	only version 1, but a version 2 may appear in a future qpdf.
+	in a future qpdf.
 
 2022-01-28  Jay Berkenbilt  <ejb@ql.org>
 

TODO

@@ -9,15 +9,6 @@
   make qpdf more contributor-friendly. Look
   https://bestpractices.coreinfrastructure.org/en
 
-* Remember for release notes: starting in qpdf 11, the default value
-  for the --json keyword will be "latest". If you are depending on
-  version 1, change your code to specify --json=1, which works
-  starting with 10.6.0.
-
-* Write up something about preparing for the PointerHolder to
-  shared_ptr migration. Clearly document the deprecations and how to
-  deal with them.
-
 Output JSON v2
 ==============
 
@@ -332,6 +323,9 @@ Other notes:
 PointerHolder to std::shared_ptr
 ================================
 
+Remember to update the smart-pointers section of the manual in
+design.rst.
+
 Once all deprecation warnings are cleared up (changing getPointer() to
 get() and getRefcount() to use_count()), the only real issues are that
 implicit assignment of a pointer to a shared_ptr doesn't work while it

cSpell.json

@@ -74,6 +74,7 @@
     "cxxflags",
     "cygwin",
     "dctdecode",
+    "decltype",
     "decrypter",
     "deduplicating",
     "deps",

job.sums

@@ -14,4 +14,4 @@ libqpdf/qpdf/auto_job_json_decl.hh c5e3fd38a3b0c569eb0c6b4c60953a09cd6bc7d3361a3
 libqpdf/qpdf/auto_job_json_init.hh b070350d304d137ba594c1ba40b373137e8459735f04b8ca0f8a2ffd1908c69e
 libqpdf/qpdf/auto_job_schema.hh 18a3780671d95224cb9a27dcac627c421cae509d59f33a63e6bda0ab53cce923
 manual/_ext/qpdf.py e9ac9d6c70642a3d29281ee5ad92ae2422dee8be9306fb8a0bc9dba0ed5e28f3
-manual/cli.rst 3746df6c4f115387cca0d921f25619a6b8407fc10b0e4c9dcf40b0b1656c6f8a
+manual/cli.rst 2dd5e5a9c0440aea65ed0a2bf6239aa6662afdb463224aafdc116a8a676dbc20

manual/cli.rst

@@ -3154,7 +3154,10 @@ Related Options
    supported value is ``1``, but it's possible that a new JSON output
    version will be added in a future version. You can also specify
    ``latest`` to use the latest JSON version. For backward
-   compatibility, the default value is ``1``. Use the
+   compatibility, the default value will remain ``1`` until qpdf
+   version 11, after which point it will become ``latest``. In all
+   case, you can tell what version of the JSON output you have from
+   the ``"version"`` key in the output. Use the
    :qpdf:ref:`--json-help` option to get a description of the JSON
    object.
 

manual/design.rst

@@ -745,3 +745,221 @@ C API object handle methods returned error codes like the other methods
 and set return values in passed-in pointers, but this would complicate
 both the implementation and the use of the library for a case that is
 actually quite rare and largely avoidable.
+
+.. _smart-pointers:
+
+Smart Pointers
+--------------
+
+This section describes changes to the use of smart pointers in qpdf in
+versions 10.6.0 and 11.0.0.
+
+Starting in qpdf 11, ``PointerHolder`` will be replaced with
+``std::shared_ptr`` in qpdf's public API. A backward-compatible
+``PointerHolder`` will be provided that should make it possible for
+most code to remain unchanged. This new ``PointerHolder`` will be
+marked deprecated but will provide a way to suppress the deprecation
+warnings. Code that works with containers of ``PointerHolder`` may
+have to be modified, though no qpdf interfaces do this.
+
+The remainder of this section describes how to prepare if you want to
+eliminate ``PointerHolder`` from your code or what to do if you want
+to stick with the old interfaces.
+
+Changes in 10.6.0
+~~~~~~~~~~~~~~~~~
+
+In qpdf 10.6.0, two ``PointerHolder`` methods have been deprecated and
+replaced with methods that are compatible with ``std::shared_ptr``:
+
+- ``getPointer()`` -- use ``get()`` instead
+
+- ``getRefcount()`` -- use ``use_count()`` instead
+
+If you build your code with deprecation warnings enabled and you want
+to suppress these deprecation warnings for now, you can ``#define
+NO_POINTERHOLDER_DEPRECATION`` before including any qpdf header files.
+It may be possible to leave it this way long-term to facilitate
+supporting older versions of qpdf without conditional compilation.
+
+``PointerHolder`` has had a long-standing bug: a ``const
+PointerHolder<T>`` would only provide a ``T const*`` with its
+``getPointer`` method. This is incorrect and is now how standard C++
+smart pointers or regular pointers behave. The correct semantics
+would be that a ``const PointerHolder<T>`` would not accept a new
+pointer after being created but would still allow you to modify the
+item being pointed to. If you don't want to mutate the thing it points
+to, use ``PointerHolder<T const>`` instead. The new ``get()`` method
+behaves correctly. It is therefore not exactly the same as
+``getPointer()``, but it does behave the way ``get()`` behaves with
+``std::shared_ptr``. This shouldn't make any difference to any
+correctly written code.
+
+
+How to Prepare
+~~~~~~~~~~~~~~
+
+If you don't need to support versions of qpdf prior to 10.6, you can
+just replace all occurrences of ``getPointer()`` with ``get()`` and
+all occurrences of ``getRefcount()`` with ``use_count()``. That's
+about all you will be able to do prior to qpdf 11.
+
+If you need to support older versions, you have two choices:
+
+- ``#define NO_POINTERHOLDER_DEPRECATION`` and leave everything the
+  way it was. You can just wait until qpdf 11.
+
+- Write code that uses ``get()`` but falls back to ``getPointer()`` if
+  ``QPDF_MAJOR_VERSION`` is not defined. The symbols
+  ``QPDF_MAJOR_VERSION``, ``QPDF_MINOR_VERSION``, and
+  ``QPDF_PATCH_VERSION`` were introduced with 10.6.0, so just checking
+  for whether ``QPDF_MAJOR_VERSION`` is defined is sufficient for
+  telling if you're running a version before 10.6.0. If you do this,
+  once qpdf 11 comes out, you will already know all the places that
+  have to be handled specially.
+
+If you are somehow relying on the fact that a ``const
+PointerHolder<T>`` always gave back a ``T const*`` and are
+dereferencing a ``const PointerHolder<T>`` to call methods that only
+have ``const`` versions in ``T``, you may have to change from
+``const PointerHolder<T>`` to ``PointerHolder<T const>``. This won't
+be an issue for anything in the qpdf API, and if you are using qpdf
+``PointerHolder`` objects for any other reason, you should just
+replace them with ``std::shared_ptr``.
+
+What to Expect
+~~~~~~~~~~~~~~
+
+Note: if you are reading this in the 10.6 manual and 11 is out, you
+should read it in the manual for qpdf 11 instead. Some early tests
+have been done to try to ensure the accuracy of this information, but
+it may change once the work is actually completed.
+
+When ``PointerHolder`` disappears from qpdf's API in qpdf 11, you will
+have a few options:
+
+- Use the new ``PointerHolder``, which is derived from
+  ``std::shared_ptr`` and which has methods to make it
+  interchangeable. For things that use ``PointerHolder<T>`` directly,
+  this should "just work," though you will have to ``#define
+  NO_POINTERHOLDER_DEPRECATION`` if you don't want deprecation
+  warnings.
+
+- Replace all uses of ``PointerHolder<T>`` with ``std::shared_ptr<T>``
+  and deal with the required changes, outlined below. This is the
+  recommended course of action. You will need conditional compilation
+  if you want to simultaneously support order code. Stay tuned for the
+  qpdf 11 documentation for specifics.
+
+While ``PointerHolder<T>`` and ``std::shared_ptr<T>`` will be mutually
+assignable and convertible, this does not apply to containers of those
+objects. The qpdf API doesn't have any containers of
+``PointerHolder``, so this would have to be in your own code. You can
+prepare yourself for the change by using ``auto`` and ``decltype``
+whenever possible so that a change to the underlying type of something
+won't require source changes.
+
+Required Changes in qpdf 11
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section describes unavoidable changes when replacing
+``PointerHolder`` with ``std::shared_ptr`` rather than continuing to
+use the backward compatible API. Nothing here is needed (or can be
+done) prior to qpdf 11, so consider this to be a preview.
+
+- Change ``getPointer`` to ``get`` and ``getRefcount`` to
+  ``use_count`` as above. If your starting point is no deprecation
+  warnings with qpdf 10.6, this will already be true.
+
+- Array allocations will have to be rewritten.
+
+  To allocate a ``PointerHolder`` to an array:
+
+  .. code-block:: c++
+
+     PointerHolder<X> p(true, new X[n]);
+
+  To allocate a ``std::shared_ptr`` to an array:
+
+  .. code-block:: c++
+
+     auto p = std::shared_ptr<X>(new X[n], std::default_delete<X[]>());
+
+  To allocate a ``std::unique_ptr`` to an array:
+
+  .. code-block:: c++
+
+     auto p = std::make_unique<X[]>(n);
+     // or
+     auto p = std::unique_ptr<X[]>(new X[n]);
+
+  The second form may be needed if ``X`` has a private constructor
+  from this context.
+
+  C++-17 has a better way to allocate ``std::shared_ptr`` to an array,
+  but qpdf is still allowing C++-14 to be used. You can use whatever
+  method to handle shared arrays that is supported in your
+  environment. There are no shared arrays in qpdf's public API except
+  for some ``QUtil`` helper methods that are not essential for use of
+  qpdf features.
+
+- ``PointerHolder<T>`` can have plain pointers directly assigned to
+  it, while ``std::shared_ptr<T>`` cannot. This makes code like this
+  possible:
+
+  .. code-block:: c++
+
+     PointerHolder<X> x_p;
+     X* x = new X();
+     x_p = x;
+
+  It also makes it possible to pass a plain pointer to a function
+  expecting a ``PointerHolder``, thereby transferring "ownership" of
+  the pointer into the function.
+
+  Code like that is a risky because you can leak memory if an
+  exception is thrown between creation of the X and assignment of it
+  into the ``PointerHolder``. In any case, ``std::shared_ptr`` does
+  not allow that, so you need one of these instead:
+
+  .. code-block:: c++
+
+     auto x_p = std::make_shared<X>();
+     X* x = x_p.get();
+     // or, less safe, but closer:
+     std::shared_ptr<X> x_p;
+     X* x = new X();
+     x_p = std::shared_ptr<X>(x);
+
+  Also, code like this:
+
+  .. code-block:: c++
+
+     PointerHolder<Base> base_p;
+     Derived* derived = new Derived();
+     base_p = derived;
+
+  needs to be replaced with something like this instead:
+
+  .. code-block:: c++
+
+     std::shared_ptr<Base> base_p;
+     Derived* derived = new Derived();
+     base_p = std::shared_ptr<Base>(derived);
+
+Historical Background
+~~~~~~~~~~~~~~~~~~~~~
+
+Since its inception, the qpdf library used its own smart pointer
+class, ``PointerHolder``. The ``PointerHolder`` class was originally
+created long before ``std::shared_ptr`` existed, and qpdf itself
+didn't start requiring a C++-11 compiler version 9.1.0 released in
+late 2019.
+
+``PointerHolder`` is a reference-counted smart pointer with semantics
+almost identical to ``std::shared_ptr`` except that it is not
+thread-safe. It has a few interface differences that prevent
+``std::shared_ptr`` from being a drop-in replacement. However, given
+the value of using standard library smart pointers, qpdf is taking the
+plunge for version 11 and switching over to standard library smart
+pointers.

manual/qpdf-job.rst

@@ -14,7 +14,10 @@ executable is available from inside the C++ library using the
 
   - Use from the C++ API with ``QPDFJob::initializeFromArgv``
 
-  - Use from the C API with ``qpdfjob_run_from_argv`` from :file:`qpdfjob-c.h`
+  - Use from the C API with ``qpdfjob_run_from_argv`` from
+    :file:`qpdfjob-c.h`. If you are calling from a Windows-style main
+    and have an argv array of ``wchar_t``, you can use
+    ``qpdfjob_run_from_wide_argv``.
 
 - The job JSON file format
 
@@ -135,6 +138,13 @@ C++ code:
           return 0;
       }
 
+Note the ``QPDFUsage`` exception above. This is thrown whenever a
+configuration error occurs. These exactly correspond to usage messages
+issued by the :command:`qpdf` CLI for things like omitting an output
+file, specifying `--pages` multiple times, or other invalid
+combinations of options. ``QPDFUsage`` is thrown by the argv and JSON
+interfaces as well as the native ``QPDFJob`` interface.
+
 It is also possible to mix and match command-line options and JSON
 from the CLI. For example, you could create a file called
 :file:`my-options.json` containing the following:

manual/release-notes.rst

@@ -6,6 +6,187 @@ Release Notes
 For a detailed list of changes, please see the file
 :file:`ChangeLog` in the source distribution.
 
+10.6.0: XXX
+  - Deprecations/future replacement of ``PointerHolder``
+
+    The next major release of qpdf will replace ``PointerHolder`` with
+    ``std::shared_ptr`` across all of qpdf's public API. In
+    preparation for this change, the following ``PointerHolder``
+    methods have been deprecated in favor of interfaces that more
+    closely match ``std::shared_ptr``:
+
+    - ``getPointer()`` -- use ``get()`` instead; this also fixes
+      ``const`` semantics as discussed in
+      :file:`include/qpdf/PointerHolder.hh`.
+
+    - ``getRefcount()`` -- use ``use_count()`` instead
+
+    If you build your code with deprecation warnings enabled and you
+    want to suppress these deprecation warnings for now, you can
+    ``#define NO_POINTERHOLDER_DEPRECATION`` before including any qpdf
+    header files. Code that does this will *require no changes* prior
+    to qpdf 11 and may or may not require changes after qpdf 11.
+
+    For a detailed discussion of this change and how to prepare for
+    it, see :ref:`smart-pointers`.
+
+  - Preparation for a new JSON output version
+
+    - The :qpdf:ref:`--json` option takes an optional parameter
+      indicating the version of the JSON output. At present, there is
+      only one JSON version (``1``), but there are plans for an
+      updated version in a coming release. Until the release of qpdf
+      11, the default value of ``--json`` is ``1`` for compatibility.
+      Once qpdf 11 is out, the default version will be ``latest``. If
+      you are depending on the exact format of ``--json`` for code,
+      you should start using ``--json=1`` in preparation.
+
+  - New QPDFJob API exposes CLI functionality
+
+    Prior to qpdf 10.6, a lot of the functionality implemented by the
+    qpdf CLI executable was built into the executable itself and not
+    available from the library. qpdf 10.6 introduces a new object,
+    ``QPDFJob``, that exposes all of the command-line functionality.
+    This includes a native ``QPDFJob`` API with fluent interfaces that
+    mirror the command-line syntax, a JSON syntax for specifying the
+    equivalent of a command-line invocation, and the ability to run a
+    qpdf "job" by passing a null-terminated array of qpdf command-line
+    options. The command-line argument array and JSON methods of
+    invoking ``QPDFJob`` are also exposed to the C API. For details,
+    see :ref:`qpdf-job`.
+
+  - Other Library Enhancements
+
+    - New ``QPDFObjectHandle`` literal syntax using C++'s user-defined
+      literal syntax. You can use
+
+      .. code-block:: c++
+
+         auto oh = "<</Some (valid) /PDF (object)>>"_qpdf;
+
+      to create a QPDFObjectHandle. It is a shorthand for
+      ``QPDFObjectHandle::parse``.
+
+    - Preprocessor symbols ``QPDF_MAJOR_VERSION``,
+      ``QPDF_MINOR_VERSION``, and ``QPDF_PATCH_VERSION`` are now
+      available and can be used to make it easier to write code that
+      supports multiple versions of qpdf. You don't have to include
+      any new header files to get these, which makes it possible to
+      write code like this:
+
+      .. code-block:: c++
+
+         #if !defined(QPDF_MAJOR_VERSION) || QPDF_MAJOR_VERSION < 11
+             // do something using qpdf 10 or older API
+         #else
+             // do something using qpdf 11 or newer API
+         #endif
+
+      Since this was introduced only in qpdf version 10.6.0, testing
+      for an undefined value of ``QPDF_MAJOR_VERSION`` is equivalent
+      to detecting a version prior to 10.6.0.
+
+      The symbol ``QPDF_VERSION`` is also defined as a string
+      containing the same version number that is returned by
+      ``QPDF::QPDFVersion``. Note that ``QPDF_VERSION`` may differ
+      from ``QPDF::QPDFVersion()`` if your header files and library
+      are out of sync with each other.
+
+    - The method ``QPDF::QPDFVersion`` and corresponding C API call
+      ``qpdf_get_qpdf_version`` are now both guaranteed to return a
+      reference (or pointer) to a static string, so you don't have to
+      copy these if you are using them in your software. They have
+      always returned static values. Now the fact that they return
+      static values is part of the API contract and can be safely
+      relied upon.
+
+    - New accessor methods for ``QPDFObjectHandle``. In addition to
+      the traditional ones, such as ``getIntValue``, ``getName``,
+      etc., there are a family of new accessors whose names are of the
+      form ``getValueAsX``. The difference in behavior is as follows:
+
+      - The older accessor methods, which will continue to be
+        supported, return the value of the object if it is the
+        expected type. Otherwise, they return a fallback value and
+        issue a warning.
+
+      - The newer accessor methods return a boolean indicating whether
+        or not the object is of the expected type. If it is, a
+        reference of the correct type is returned.
+
+      In many cases, the new interfaces will enable more compact code
+      and will also never generate type warnings. Thanks to M. Holger
+      for contributing these accessors. Search for ``getValueAs`` in
+      :file:`include/qpdf/QPDFObjectHandle.hh` for a complete list.
+
+      These are also exposed in the C API in functions whose names
+      start with ``qpdf_oh_get_value_as``.
+
+    - New convenience methods in ``QPDFObjectHandle``:
+      ``isDictionaryOfType``, ``isStreamOfType``, and
+      ``isNameAndEquals`` allow more compact querying of dictionaries.
+      Also added to the C API: ``qpdf_oh_is_dictionary_of_type`` and
+      ``qpdf_oh_is_name_and_equals``. Thanks to M. Holger for the
+      contribution.
+
+    - New functions added to ``QUtil``: ``make_shared_cstr`` and
+      ``make_unique_cstr`` copy ``std::string`` to
+      ``std::shared_ptr<char>`` and ``std::unique_ptr<char[]>``. These
+      are alternatives to the existing ``QUtil::copy_string`` function
+      which offer other ways to get a C string with safer memory
+      management.
+
+    - New function ``QUtil::file_can_be_opened`` tests to see whether
+      a file can actually be opened by attempting to open it and close
+      it again.
+
+    - There is a new version of ``QUtil::call_main_from_wmain`` that
+      takes a ``const`` argv array and calls a main that takes a
+      ``const`` argv array.
+
+    - ``QPDF::emptyPDF`` has been exposed to the C API as
+      ``qpdf_empty_pdf``. This makes it possible to create PDF from
+      scratch with the C API.
+
+    - New C API functions ``qpdf_oh_get_binary_utf8_value`` and
+      ``qpdf_oh_new_binary_unicode_string`` take length parameters,
+      which makes it possible to handle UTF-8-encoded C strings with
+      embedded NUL characters. Thanks to M. Holger for the
+      contribution.
+
+    - The ``JSON`` object in the qpdf library has been enhanced to
+      include a parser and the ability to get values out of the
+      ``JSON`` object. Previously it was a write-only interface. Even
+      so, qpdf's ``JSON`` object is not intended to be a
+      general-purpose JSON implementation as discussed in
+      :file:`include/qpdf/JSON.hh`.
+
+    - The ``JSON`` object's "schema" checking functionality now allows
+      for optional keys. Note that this "schema" functionality doesn't
+      conform to any type of standard. It's just there to help with
+      error reporting with qpdf's own JSON support.
+
+  - Documentation Enhancements
+
+    - Documentation for the command-line tool has been completely
+      rewritten. This includes a top-to-bottom rewrite of :ref:`using`
+      in the manual. Command-line arguments are now indexed, and
+      internal links can appear to them within the documentation.
+
+    - The output of ``qpdf --help`` is generated from the manual and
+      is divided into help topics that parallel the sections of the
+      manual. When you run ``qpdf --help``, instead of getting a Great
+      Wall of Text, you are given basic usage information and a list
+      of help topics. It is possible to request help for any
+      individual topic or any specific command-line option, or you can
+      get a dump of all available help text. The manual continues to
+      contain a greater level of detail and more examples.
+
+  - Bug Fixes
+
+    - Some characters were not correctly translated from PDF doc
+      encoding to Unicode.
+
 10.5.0: December 21, 2021
   - Packaging changes