ROUTINE DEVELOPMENT **Remember to check pull requests as well as issues in github.** Default: ./configure CXX="g++ --std=c++14" --enable-werror --enable-maintainer-mode Debugging: ./configure CXX="g++ --std=c++14" CFLAGS="-g" CXXFLAGS="-g" \ --enable-werror --disable-shared --enable-maintainer-mode Profiling: ./configure CXX="g++ --std=c++14" CFLAGS="-g -pg" CXXFLAGS="-g -pg" \ LDFLAGS="-pg" --enable-werror --disable-shared --enable-maintainer-mode Then run `gprof gmon.out`. Note that gmon.out is not cumulative. Memory checks: ./configure CFLAGS="-fsanitize=address -fsanitize=undefined -g" \ CXXFLAGS="-fsanitize=address -fsanitize=undefined -g" \ LDFLAGS="-fsanitize=address -fsanitize=undefined" \ CC=clang CXX="clang++ --std=c++14" \ --enable-werror --disable-shared --enable-maintainer-mode VERSIONS * The version number on the main branch is whatever the version would be if the top of the branch were released. If the most recent release is version a.b.c, then * If there are any ABI-breaking changes since the last release, main's version is a+1.0.0 * Else if there is any new public API, main's version is a.b+1.0 * Else if there are any code changes, main's version is a.b.c+1 * Whenever we bump the version number, bump shared library versions as well. * Every released major/minor version has an a.b branch which is used primarily for documentation but could potentially be used to create a new patch release after main has moved on. We don't do that as a rule, but there's no reason we couldn't do it if main had unreleased ABI/API changes that were still in flux and an important bug fix was needed on the most recent release. In that case, a release can be cut from a release branch and then either main can be rebased from there or the changes can be merged back, depending on the amount of drift. CHECKING DOCS ON readthedocs To check docs on readthedocs.io without running all of CI, push to the doc-check branch. Then visit https://qpdf.readthedocs.io/en/doc-check/ Building docs from pull requests is also enabled. GOOGLE OSS-FUZZ * See ../misc/fuzz (not in repo) for unfixed, downloaded fuzz test cases * qpdf project: https://github.com/google/oss-fuzz/tree/master/projects/qpdf * Adding new test cases: download the file from oss-fuzz and drop it in fuzz/qpdf_extra/issue-number.fuzz. When ready to include it, add to fuzz/CMakeLists.txt. Until ready to use, the file can be stored anywhere, and the absolute path can be passed to the reproduction code as described below. * To test locally, see https://github.com/google/oss-fuzz/tree/master/docs/, especially new_project_guide.md. Summary: Clone the oss-fuzz project. From the root directory of the repository: Add `-e GITHUB_FORK=fork -e GITHUB_BRANCH=branch` to build_fuzzers from a qpdf fork/branch rather than qpdf/main. python3 infra/helper.py build_image --pull qpdf python3 infra/helper.py build_fuzzers [ --sanitizer memory|undefined|address ] qpdf python3 infra/helper.py check_build qpdf python3 infra/helper.py build_fuzzers --sanitizer coverage qpdf python3 infra/helper.py coverage qpdf To reproduce a test case, build with the correct sanitizer, then run python3 infra/helper.py reproduce qpdf testcase where fuzzer is the fuzzer used in the crash. The fuzzer is in build/out/qpdf. It can be run with a directory as an argument to run against files in a directory. You can use qpdf_fuzzer -merge=1 cur new >& /dev/null& to add any files from new into cur if they increase coverage. You need to do this with the coverage build (the one with --sanitizer coverage) * General documentation: http://libfuzzer.info * Build status: https://oss-fuzz-build-logs.storage.googleapis.com/index.html * Project status: https://oss-fuzz.com/ (private -- log in with Google account) * Latest corpus: gs://qpdf-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/qpdf_fuzzer/latest.zip CODING RULES * In a source file, include the header file that declares the source class first followed by a blank line. If a config file is needed first, put a blank line between that and the header followed by another blank line. This assures that each header file is included first at least once, thereby ensuring that it explicitly includes all the headers it needs, which in turn alleviates lots of header ordering problems. The blank line ensures that formatters don't mess this up by resorting the headers. * Avoid atoi. Use QUtil::string_to_int instead. It does overflow/underflow checking. * Avoid certain functions that tend to be macros or create compilation errors on some platforms. Known cases: strcasecmp, abs. Avoid min and max. If needed, std::min and std::max are okay to use in C++ code with included. * Remember to avoid using `operator[]` with `std::string` or `std::vector`. Instead, use `at()`. See README-hardening.md for details. * Use QIntC for type conversions -- see casting policy in docs. * Remember to imbue ostringstreams with std::locale::classic() before outputting numbers. This protects against the user's global locale altering otherwise deterministic values. (See github issue #459.) One could argue that error messages containing numbers should respect the user's locale, but I think it's more important for output to be consistent, since the messages in question are not really targeted at the end user. * Use QPDF_DLL on all methods that are to be exported in the shared library/DLL. Use QPDF_DLL_CLASS for all classes whose type information is needed. This is important for exception classes and it seems also for classes that are intended to be subclassed across the shared library boundary. * Put private member variables in PointerHolder for all public classes. Remember to use QPDF_DLL on ~Members(). Exception: indirection through PointerHolder is expensive, so don't do it for classes that are copied a lot, like QPDFObjectHandle and QPDFObject. * Traversal of objects is expensive. It's worth adding some complexity to avoid needless traversals of objects. * Avoid attaching too much metadata to objects and object handles since those have to get copied around a lot. HOW TO ADD A COMMAND-LINE ARGUMENT QPDFJob is documented in three places: * This section provides a quick reminder for how to add a command-line argument * generate_auto_job has a detailed explanation about how QPDFJob and generate_auto_job work together * The manual ("QPDFJob Design" in qpdf-job.rst) discusses the design approach, rationale, and evolution of QPDFJob. Command-line arguments are closely coupled with QPDFJob. To add a new command-line argument, add the option to the appropriate table in job.yml. This will automatically declare a method in the private ArgParser class in QPDFJob_argv.cc which you have to implement. The implementation should make calls to methods in QPDFJob via its Config classes. Then, add the same option to either the no-json section of job.yml if it is to be excluded from the job json structure, or add it under the json structure to the place where it should appear in the json structure. In most cases, adding a new option will automatically declare and call the appropriate Config method, which you then have to implement. If you need a manual handler, you have to declare the option as manual in job.yml and implement the handler yourself, though the automatically generated code will declare it for you. The build will fail until the new option is documented in manual/cli.rst. To do that, create documentation for the option by adding a ".. qpdf:option::" directive followed by a magic help comment as described at the top of manual/cli.rst. Put this in the correct help topic. Help topics roughly correspond with sections in that chapter and are created using a special ".. help-topic" comment. Follow the example of other options for style. When done, the following should happen: * qpdf --new-option should work as expected * qpdf --help=--new-option should show the help from the comment in cli.rst * qpdf --help=topic should list --new-option for the correct topic * --new-option should appear in the manual * --new-option should be in the command-line option index in the manual * A Config method (in Config or one of the other Config classes in QPDFJob) should exist that corresponds to the command-line flag * The job JSON file should have a new key in the schema corresponding to the new option RELEASE PREPARATION * Each year, update copyright notices. This will find all relevant places (assuming current copyright is from last year): git --no-pager grep -i -n -P "copyright.*$(expr $(date +%Y) - 1).*berkenbilt" Also update the copyright in these places: * debian package -- search for copyright.*berkenbilt in debian/copyright * qtest-driver, TestDriver.pm in qtest source Copyright last updated: 2022. * Take a look at "External Libraries" in TODO to see if we need to make any changes. There is still some automation work left to do, so handling external-libs releases is still manual. See also README-maintainer in external-libs. * Check for open fuzz crashes at https://oss-fuzz.com * Check lgtm: https://lgtm.com/projects/g/qpdf/qpdf/?mode=list * Check all open issues and pull requests in github and the sourceforge trackers. See ~/scripts/github-issues. Don't forget pull requests. Note: If the location for reporting issues changes, do a careful check of documentation and code to make sure any comments that include the issue creation URL are updated. * Check `TODO` file to make sure all planned items for the release are done or retargeted. * Check work `qpdf` project for private issues * Run a spelling checker over the source code to catch errors in variable names, strings, and comments. make spell CLEAN=1 This uses cspell. Install with `npm install -g cspell`. The output of cspell is suitable for use with `M-x grep` in emacs. Add exceptions to cSpell.json. * If needed, run large file and image comparison tests. Configure options: --enable-test-compare-images --with-large-file-test-path=/path For Windows, use a Windows style path, not an MSYS path for large files. * Test with clang. Pass `CC=clang CXX=clang++` to `./configure`. (Done in CI). * Test with newer version of gcc if available. * Test 32-bit. Pass `CC=i686-linux-gnu-gcc CXX=i686-linux-gnu-g++` to `./configure`. (Done in CI.) * Test build on a mac. (Done in CI.) * Test with address sanitizer as described above. (Done in CI.) * A small handful of additional files have been taken from autotools programs. These should probably be updated from time to time. * `config.guess`, `config.sub`, `ltmain.sh`, and the `m4` directory: these were created by running `libtoolize -c`. To update, run `libtoolize -f -c` or remove the files and rerun `libtoolize`. For `config.guess` and `config.sub`, search for "latest" in the files, and follow directions for updating them. * Other files copied as indicated: ``` cp /usr/share/automake-1.16/install-sh . cp /usr/share/automake-1.16/mkinstalldirs . cp /usr/share/aclocal/pkg.m4 m4 ``` The entire contents of the `m4` directory came from `libtool.m4`. If we had some additional local parts, we could also add those to the `m4` directory. In order for this to work, it is necessary to run `aclocal -I m4` before running `autoheader` and `autoconf`. The `autogen.sh` script handles this. * If any interfaces were added or changed, check C API to see whether changes are appropriate there as well. If necessary, review the casting policy in the manual, and ensure that integer types are properly handled with QIntC or the appropriate cast. Remember to ensure that any exceptions thrown by the library are caught and converted. See `trap_errors` in qpdf-c.cc. * Double check versions and shared library details. They should already be up to date in the code. * Increment shared library version information as needed (`LT_CURRENT` in `configure.ac`) * Make sure version numbers are consistent in the following locations: * configure.ac * include/qpdf/DLL.h * manual/conf.py `make_dist` verifies this consistency. * Run ./autogen.sh * Update release notes in manual. Look at diffs and ChangeLog. Update release date in `manual/release-notes.rst`. * Add a release entry to ChangeLog: "x.y.z: release" * Commit title: "Prepare x.y.z release" * Performance test is included with binary compatibility steps. Even if releasing a new major release and not doing binary compatibility testing, do performance testing. * Test for performance and binary compatibility: * Check out the last release * ./configure --enable-werror && make -j$(nproc) * Check out the current version * ./performance_check | tee -a /tmp/perf * ./configure --enable-werror && make -j$(nproc) build_libqpdf * Checkout the last release * make -k check NO_REBUILD=1 (some failures are normal -- looking for binary compatibility) * Check out the current version * make -j$(nproc) * ./performance_check | tee -a /tmp/perf * Run pikepdf's test suite. Do this in a separate shell. cd ...qpdf-source-tree... export QPDF_SOURCE_TREE=$PWD export LD_LIBRARY_PATH=$QPDF_SOURCE_TREE/libqpdf/build/.libs cd /tmp/z git clone git@github.com:pikepdf/pikepdf virtualenv v source v/bin/activate cd pikepdf pip3 install --upgrade pip pip3 install '.[test]' rehash pip3 install . pytest -n auto CREATING A RELEASE * Push to main. This will create an artifact called distribution which will contain all the distribution files. Download these, verify the checksums from the job output, rename to remove -ci from the names, and copy to the release archive area. * Sign the source distribution: version=x.y.z gpg --detach-sign --armor qpdf-$version.tar.gz * Build and test the debian package * Add a calendar reminder to check the status of the debian package to make sure it is transitioning properly and to resolve any issues. * Sign the releases. The release archive area should contain the Windows binaries, the AppImage, the source tarball, and the source tarball signature. \rm -f *.sha256 files=(*) sha256sum ${files[*]} >| qpdf-$version.sha256 gpg --clearsign --armor qpdf-$version.sha256 mv qpdf-$version.sha256.asc qpdf-$version.sha256 chmod 444 * chmod 555 *.AppImage * When creating releases on github and sourceforge, remember to copy `README-what-to-download.md` separately onto the download area if needed. * Ensure that the main branch has been pushed to github. The rev-parse command below should show the same commit hash for all its arguments. Create and push a signed tag. This should be run with HEAD pointing to the tip of main. git rev-parse qpdf/main @ git tag -s release-qpdf-$version @ -m"qpdf $version" git push qpdf release-qpdf-$version * Update documentation branches git push qpdf @:$(echo $version | sed -E 's/\.[^\.]+$//') git push qpdf @:stable * If this is an x.y.0 release, visit https://readthedocs.org/projects/qpdf/versions/ (log in with github), and activate the latest major/minor version * Create a github release after pushing the tag. `gcurl` is an alias that includes the auth token. # Create release GITHUB_TOKEN=$(qdata-show cred github-token) function gcurl() { curl -H "Authorization: token $GITHUB_TOKEN" ${1+"$@"}; } url=$(gcurl -s -XPOST https://api.github.com/repos/qpdf/qpdf/releases -d'{"tag_name": "release-qpdf-'$version'", "name": "qpdf '$version'", "draft": true}' | jq -r '.url') # Get upload url upload_url=$(gcurl -s $url | jq -r '.upload_url' | sed -E -e 's/\{.*\}//') echo $upload_url # Upload all the files. You can add a label attribute too, which # overrides the name. for i in *; do mime=$(file -b --mime-type $i) gcurl -H "Content-Type: $mime" --data-binary @$i "$upload_url?name=$i" done If needed, go onto github and make any manual updates such as indicating a pre-release, adding release notes, etc. Template for release notes: ``` This is qpdf version x.y.z. (Brief description) For a full list of changes from previous releases, please see the [release notes](https://qpdf.readthedocs.io/en/stable/release-notes.html). See also [README-what-to-download](./README-what-to-download.md) for details about the available source and binary distributions. ``` # Publish release gcurl -XPOST $url -d'{"draft": false}' * Upload files to sourceforge. rsync -vrlcO ./ jay_berkenbilt,qpdf@frs.sourceforge.net:/home/frs/project/q/qp/qpdf/qpdf/$version/ * On sourceforge, make the source package the default for all but Windows, and make the 32-bit mingw build the default for Windows. * Publish a news item manually on sourceforge. * Upload the debian package and Ubuntu ppa backports. * Email the qpdf-announce list. OTHER NOTES For local iteration on the AppImage generation, it works to just ./build-scripts/build-appimage and get the resulting AppImage from the distribution directory. You can also pass -e SKIP_TESTS=1 build-appimage, which passes it along to to docker, to skip the test suite, which useful for rapid iteration. GENERAL BUILD STUFF QPDF uses autoconf and libtool but does not use automake. The only files distributed with the qpdf source distribution that are not controlled are `configure`, `libqpdf/qpdf/qpdf-config.h.in`, `aclocal.m4`, and some documentation. See above for the steps required to prepare a source distribution. If building or editing documentation, configure with `--enable-doc-maintenance`. This will ensure that all tools or files required to validate and build documentation are available. If you want to run `make maintainer-clean` or `make distclean` and you haven't run `./configure`, you can pass `CLEAN=1` to make on the command line to prevent it from complaining about configure not having been run. If you want to run checks without rerunning the build, pass `NO_REBUILD=1` to make. This can be useful for special testing scenarios such as validation of memory fixes or binary compatibility. LOCAL WINDOWS TESTING PROCEDURE This is what I do for routine testing on Windows. From Windows, git clone from my Linux clone, and unzip `external-libs`. Look at `make_windows_releases`. Set up path the same way and run whichever `./config-*` is appropriate for whichever compiler I need to test with. Start one of the Visual Studio native compiler shells, and from there, run one of the msys shells. The Visual Studio step is not necessary if just building with mingw. DOCS ON readthedocs.org * Registered for an account at readthedocs.org with my github account * Project page: https://readthedocs.org/projects/qpdf/ * Docs: https://qpdf.readthedocs.io/ * Admin -> Settings * Set project home page * Advanced * Show version warning * Default version: stable * Email Notifications: set email address for build failures At this time, there is nothing in .github/workflows to support this. It's all set up as an integration directly between github and readthedocs. The way readthedocs.org does stable and versions doesn't exactly work for qpdf. My tagging convention is different from what they expect, and I don't need versions for every point release. I have the following branching strategy to support docs: * x.y -- points to the latest x.y.z release * stable -- points to the latest release The release process includes updating the approach branches and activating versions.