2023 PhotoStructure release notes

This is a detailed list of changes in each version.

23.5.0-prealpha.1 #

(to be released)

(This version’s contents had previously been listed as v2.1.0-alpha.8 , but we’re switching to CalVer, using scheme YY.MM.MINOR .)

✨ PhotoStructure no longer “fails fast.” #

What’s that mean?

PhotoStructure will try to always stay up and running, even if your library isn’t available, or something’s amiss, like a misconfiguration or something broken on the system.

If anything prevents your library from being open, PhotoStructure will automatically redirect to a new /health page that list several handfuls of health checks to help people diagnose what’s amiss, and in some cases, buttons that can attempt to repair what’s wrong.

PhotoStructure’s new health check page

PhotoStructure’s new health check page

This means people running PhotoStructure for Docker without reading the instructions will be presented with a friendly screen with direct links to the relevant documentation.

This change also meant I could put the /about page on a diet–it only holds fairly cheap and cached content now, so it shouldn’t disconcertinly hang anymore (prior versions ran several “health checks” that were run synchronously whenever the about page was requested).

This change also means PhotoStructure stays up even if your library hard drive gets periodically disconnected.

Read more about this change on the forum.

✨ SQLite improvements #

  • PhotoStructure now automatically figures out the best value for PS_FORCE_LOCAL_DB_REPLICA. Previously, we simply defaulted all docker installs to use a local db replica, whose implementation was problematic with prior versions. This determination is also run within a filesystem advisory lock, to prevent concurrent db setup collisions.

  • Database backups are now always taken “hot.” Prior versions required acquiring a halt-the-world mutex to prevent cold backups from causing SQLite corruption (and prior versions had some codepaths that didn’t acquire the lock durably). Prior versions could also miss copying over the -wal write-ahead log, which could also cause SQLite corruption.

  • The new db health check now validates file integrity, foreign keys, and that the schema comprehensively matches expectations for the current version.

✨ PhotoStructure for Desktops improvements #

  • The main window now preserves placement (even across screens) and dimensions between runs.

  • The View menu now has links to go back, open the log and sync reports directories, the system settings.toml, and the library settings.toml

✨ PhotoStructure for Servers improvements #

  • Docker and node editions now have a splash screen to see wth is going on at startup (without having to tail logs). (This page only shows if your library is quite large, your computer is quite slow, or a combination of both).

  • If PhotoStructure can’t open the current library, instead of crashing, a new “PhotoStructure Status” page will be shown with diagnostics to help debug what went wrong and links to how to fix it. This should be a lot more friendly (especially as a first impression) for most people.

  • PhotoStructure’s binaries and supporting files were moved from /ps/app to /opt/photostructure. This move shouldn’t impact anyone, and was made to avoid people being confused by mounting anything to /ps and hiding the entire installation.

✨/🐛/🏗️ Image deduplication improvements #

  • Dominant color extraction now uses adaptive greyscale prefiltering, iterative k-means clustering, and returns percent coverage per color. This change required a database migration (that will be applied automatically) and a library rebuild (that will be scheduled automatically). See the new dominantColorPixels, dominantColorKmeansRuns, dominantColorMergeThreshold, and dominantColorGreyThreshold settings.

  • Prior builds relied on a single (mean) image hash algorithm. This build adds two additional, novel CIELAB-based approaches (gradient diff and DCT). Having 3 different hashes dramatically helps both precision and recall.

  • Image hashes now use higher-quality resizing interpolation.

  • New settings to control correlation thresholds: minImageCoeffPct, minColorCoeffPct, imageHashFuzzyDateDelta, imageHashFuzzyDateDelta, imageHashRotationDelta, imageHashDifferentMimetypesDelta, and imageHashGreyscaleDelta.

  • When comparing two files, if either one of the files has an “imprecise” or “fuzzy” captured-at value (if the source is from the filename, inferred or from Stats), the image hash is always used, and the captured-at can be different. Disable this behavior by setting strictDeduping=true. Prior builds would skip the image hash comparison in some cases. This change will require libraries to be rebuilt when upgraded to this build.

✨/🐛 Time parsing improvements #

  • Video files are notoriously hard to get correct captured-at timezone offset values. Videos regularly encode the CreateDate tag in UTC (even when the file wasn’t captured in UTC!). This results in videos from prior versions of PhotoStructure being wrong by several hours. PhotoStructure now tries to “repair” the UTC timezone into the correct timezone by using either GPS metadata or a timezone offset inferrable via the filename. PhotoStructure also now has prioritized tag extraction: see the capturedAtTags and new capturedAtTagsFallback settings. Note that some Quicktime tags are not reliably stored as UTC, so we look for more reliable tags before resorting to these problematic tags. If you find a video doesn’t have the correct time in your PhotoStructure library, please email us an example.

  • Timezone parsing has been improved to support both IANA and ISO offset formats (both of which have been found in the wild 😠).

  • PhotoStructure handles missing timezones and differing timestamp precision more intelligently now: see minCapturedAtPrecision and fuzzyDatePrecisionCoeff for explanations.

  • Google Takeout JSON sidecar timestamps no longer (incorrectly) inherit the current system timezone.

Other improvements and bugfixes #

  • ✨ Asset file aggregation is now stricter. Previous versions of PhotoStructure attached file variations to existing assets as long as they matched any asset file associated to the asset. PhotoStructure will now aggregate new asset files only if they match all asset file variations. Set the new assetAggregation setting to union to restore prior behavior.

  • ✨ Both Alpine and Ubuntu Docker images are now available.

    • Alpine images are less than half the size of the Ubuntu-based images
    • Ubuntu’s ffmpeg package supports more video codecs
    • Ubuntu has better GPU acceleration support

  • ✨ psnet: asset file URIs now support sshfs-mounted partitions

  • ✨/📦 Node.js version 18.x LTS (long term support) is now supported on all platforms.

    Note that PhotoStructure now requires at least Node.js v14 (the oldest currently-supported version of Node.js).

  • ✨/🐛 Prior versions on Windows and Raspberry Pi on slow disk could result in invalid file lock timeouts, which could prevent some file types (like large HEIFs) from being imported. This could show up as EBUSY or ENOENT errors in your sync report or logs.

  • ✨/🐛 Some camera models (like the Galaxy S8+) can produce images that have JPEG encoding errors. Prior builds would prevent importing of these images. (Thanks for the example images, @nighthawk!)

    Handling these images required a couple changes:

    1. A new setting, imageFailOn, lets you import images that have minor encoding defects by default, but still reject images that have been truncated.

    2. Default values for the image validation patterns, validationErrorBlocklist and validationErrorAllowlist, now handle more corruption patterns.

  • 💔/🐛 The processPriority setting no longer supports AboveNormal. If your settings used this value, processPriority will resort to the default, Idle. (AboveNormal only worked if PhotoStructure was running as root (which it never should do!)

  • ✨/📦 The /site.webmanifest file is now dynamically generated, and includes a proper start_url (so every launch will pick a new seed) and defaults to display: fullscreen.

  • ✨/📦 info tool improvements:

    • Image hash comparison information, including all correlations, deltas, and thresholds, are now included for files (which may help tune imageHash*Delta settings values).

    • Dominiant colors now include friendly names and percent image coverage.

    • Limit output to only image hash metadata with the new --image-hash filter.

    • pathToLibraryAsset is now rendered for every file to help debug the assetPathnameFormat setting.

    • When given more than one file applies clustering on the entire array and will return the files provided to ARGV, grouped by asset.

    • Several additional switches were added to info to help customer support, including --read-settings, --suggested-libraries, and --child-env.

  • ✨/🐛 Files with extensions that don’t match their mimetype (say, JPEG-encoded images named image.dng, which Google Takeouts likes to do) are now imported gracefully.

  • ✨/🐛 settings.toml and .psenv files are now read correctly when BOM-encoded as UTF-8 or UTF16-LE.

  • 🐛 Glob exclusion patterns were not being applied correctly on Windows

  • 🐛 Newer linux distributions could pull in a version of heif-convert that has a buggy filename parser. PhotoStructure invokes this tool in such a way that this bug is avoided.

  • 🐛 Prior builds would cache the absence of heif-convert until restart, which caused confusion for some users. PhotoStructure will now detect newly-installed heif-convert binaries within a minute.

  • 🐛 Fixed docker :alpha, :beta, and :stable tagging (see the simplified example)

  • 🐛 Rewrote how tools (like ffmpeg, heif-convert, and jpegtran) are detected on the system. Rather than spawning which, or asking PowerShell for the binary path, we now walk $PATH looking for binaries with rx access. If $PATH is somehow truncated or invalid, we also walk some default paths (like %SYSTEMROOT% on Windows, and /usr/bin and /usr/local/bin on macOS and Linux).

    This fixes sqlite.exe not found and jpegtran.exe not found errors on Windows, and should fix SQLite backups on Windows.

  • 🐛 Fixed o.toLocal is not a function, caused if an asset file fails to extact a captured-at time. (Thanks for reporting, @pmocek!)

  • 🐛 Fix AssetFile constraint violation during the asset file cleanup in rebuilds. This could prevent library rebuilds from completing successfully.

  • 🐛 If an Apache reverse proxy closed the SSE socket, PhotoStructure would pop up a “🌩 Not connected” error. This build skips showing that error and tries to quietly restore the SSE socket when broken. See this forum post for details.

  • 🐛 PhotoStructure can now allow drives to go to sleep. It should “just work,” but to set volumeMetadataTtlMs=0 and mountpointsTtlMs=0 to force this behavior on platforms that don’t have mountpoint-change-watcher functionality. mountpointsTtlMs defaults to 0 on docker now, btw.

  • 🐛 Fixed off-center home icon on Safari (the displayPath for the root tag was [ null ], oops)

  • 🐛 Note: file picker dialogs on PhotoStructure for Desktops that use Linux Gnome can pop-under as a “feature” of Gnome. See https://github.com/electron/electron/issues/32857 for details.

  • ✨/📦 System load is now exposed in the about page and the info tool.

  • ✨/📦 File I/O was reduced a bit–permission checks now directly use the Stats object if cached, rather than requiring a separate access() I/O call.

  • ✨/📦 “Actual path” resolution on case-insensitive systems now ensure the correctly-cased pathname is used for URIs. Incorrect case could prevent cross-platform asset file correlation.

  • 📦 System load on macOS and Linux now average together both proc/cpu metrics as well as loadavg(), which should take into account

  • 📦 Control SQLite’s synchronous mode via the new dbSynchronousMode setting

  • 📦 If file copies are problematic (you’d see warn log entries to this effect), you can now force PhotoStructure to use cp -af (on macOS and Linux) or Copy-Item (on Windows) by setting the new onlyNativeFileCopy setting to true.

  • 📦 Control PowerShell child process concurrency via the new powerShellProcs setting

  • 📦 Added support for newer NEF and old KDC image formats

  • 📦 New setting twoDigitCutoffYear: sets the cutoff year after which a string encoding a year as two digits is interpreted to occur in the current century. As an example, a value of “50” would make “49” be interpreted as 1949, and “50” as 2050. See https://moment.github.io/luxon/api-docs/index.html#settingstwodigitcutoffyear for details. This defaults to 3 years in the future (modulus 100) and is updated automatically.

  • 📦 Some string handling previously used now-deprecated .substr(). PhotoStructure now uses locale-aware grapheme splitting where available, which should prevent high-unicode text from being corrupted.

  • 📦 Volume UUID files, .JSON files, and .TOML files all now support UTF-8, UTF-8-with-BOM, and UTF-16LE-with-BOM encodings

  • 📦 Added 20 new serial-to-model-name translations for recently released smartphones and cameras

  • 📦 Pulled in latest versions of Electron, sharp, node, TypeScript, ExifTool, and other third-party libraries.

  • 📦 When spinning up the photostructure/server docker image, better error messages are now emitted when /ps/library is missing.

  • 📦 Mountpoint and volume extraction is now more configurable. See the new excludedFilesystemTypes, excludedRootDirectories and excludedMountpoints settings. isExcludedMountpoint() now debug-logs why a given mountpoint is excluded, to help tune these settings.

  • 📦 Process renice-ing and management should be more efficient, as PhotoStructure now defaults to libuv and only resorts to external tooling on failure.

  • 📦 Supported file extensions and mimetypes are now defined in a single dictionary to ensure they are kept in synchronization. Due to the prior design, several more obscure file extensions (like .KDC) weren’t handled properly.

  • 📦 Removed --progress from sync (it wasn’t used, and was an unnecessary third-party dependency)

  • 📦 Most URLs in text files and emitted to stdout were wrapped in angle brackets, but some apps would interpret the trailing > was part of the URL (looking at you, UnRaid terminal), which 404ed. All wrapped URLs are just plaintext, separated with whitespace.

Prior release notes #