2023 PhotoStructure release notes
This is a detailed list of changes in each version.
Major releases have posts summarizing bigger changes. See the posts tagged with “release notes”.
New releases, starting in 2023, use “calendar versioning,” or CalVer, using scheme
Stable, released versions are recommended. See the forum post for details about alpha, beta, and stable releases.
“Pre-release” builds (those that include
beta) have not been thoroughly tested, and may not even launch.
betabuilds if you have recent backups.
If you update to an alpha or beta build and you want to downgrade to a prior version, know that older versions of PhotoStructure may not be able to open libraries created by newer versions of PhotoStructure. You will probably need to restore your library from a database backup.
(to be released)
(This version’s contents had previously been listed as
v2.1.0-alpha.8 , but we’re switching to CalVer, using scheme
✨ PhotoStructure no longer “fails fast.” #
What’s that mean?
PhotoStructure will try to always stay up and running, even if your library isn’t available, or something’s amiss, like a misconfiguration or something broken on the system.
If anything prevents your library from being open, PhotoStructure will automatically redirect to a new
/health page that list several handfuls of health checks to help people diagnose what’s amiss, and in some cases, buttons that can attempt to repair what’s wrong.
This means people running PhotoStructure for Docker without reading the instructions will be presented with a friendly screen with direct links to the relevant documentation.
This change also meant I could put the
/about page on a diet–it only holds fairly cheap and cached content now, so it shouldn’t disconcertinly hang anymore (prior versions ran several “health checks” that were run synchronously whenever the about page was requested).
This change also means PhotoStructure stays up even if your library hard drive gets periodically disconnected.
Read more about this change on the forum.
✨ SQLite improvements #
PhotoStructure now automatically figures out the best value for PS_FORCE_LOCAL_DB_REPLICA. Previously, we simply defaulted all docker installs to use a local db replica, whose implementation was problematic with prior versions. This determination is also run within a filesystem advisory lock, to prevent concurrent db setup collisions.
Database backups are now always taken “hot.” Prior versions required acquiring a halt-the-world mutex to prevent cold backups from causing SQLite corruption (and prior versions had some codepaths that didn’t acquire the lock durably). Prior versions could also miss copying over the
-walwrite-ahead log, which could also cause SQLite corruption.
The new db health check now validates file integrity, foreign keys, and that the schema comprehensively matches expectations for the current version.
✨ PhotoStructure for Desktops improvements #
The main window now preserves placement (even across screens) and dimensions between runs.
Viewmenu now has links to go back, open the log and sync reports directories, the system
settings.toml, and the library
✨ PhotoStructure for Servers improvements #
Docker and node editions now have a splash screen to see wth is going on at startup (without having to tail logs). (This page only shows if your library is quite large, your computer is quite slow, or a combination of both).
If PhotoStructure can’t open the current library, instead of crashing, a new “PhotoStructure Status” page will be shown with diagnostics to help debug what went wrong and links to how to fix it. This should be a lot more friendly (especially as a first impression) for most people.
PhotoStructure’s binaries and supporting files were moved from
/opt/photostructure. This move shouldn’t impact anyone, and was made to avoid people being confused by mounting anything to
/psand hiding the entire installation.
✨/🐛/🏗️ Image deduplication improvements #
Dominant color extraction now uses adaptive greyscale prefiltering, iterative k-means clustering, and returns percent coverage per color. This change required a database migration (that will be applied automatically) and a library rebuild (that will be scheduled automatically). See the new
Prior builds relied on a single (mean) image hash algorithm. This build adds two additional, novel CIELAB-based approaches (gradient diff and DCT). Having 3 different hashes dramatically helps both precision and recall.
Image hashes now use higher-quality resizing interpolation.
New settings to control correlation thresholds:
When comparing two files, if either one of the files has an “imprecise” or “fuzzy” captured-at value (if the source is from the filename, inferred or from
Stats), the image hash is always used, and the captured-at can be different. Disable this behavior by setting
strictDeduping=true. Prior builds would skip the image hash comparison in some cases. This change will require libraries to be rebuilt when upgraded to this build.
✨/🐛 Time parsing improvements #
Video files are notoriously hard to get correct captured-at timezone offset values. Videos regularly encode the
CreateDatetag in UTC (even when the file wasn’t captured in UTC!). This results in videos from prior versions of PhotoStructure being wrong by several hours. PhotoStructure now tries to “repair” the UTC timezone into the correct timezone by using either GPS metadata or a timezone offset inferrable via the filename. PhotoStructure also now has prioritized tag extraction: see the
capturedAtTagsFallbacksettings. Note that some Quicktime tags are not reliably stored as UTC, so we look for more reliable tags before resorting to these problematic tags. If you find a video doesn’t have the correct time in your PhotoStructure library, please email us an example.
Timezone parsing has been improved to support both IANA and ISO offset formats (both of which have been found in the wild 😠).
PhotoStructure handles missing timezones and differing timestamp precision more intelligently now: see
Google Takeout JSON sidecar timestamps no longer (incorrectly) inherit the current system timezone.
Other improvements and bugfixes #
✨ Asset file aggregation is now stricter. Previous versions of PhotoStructure attached file variations to existing assets as long as they matched any asset file associated to the asset. PhotoStructure will now aggregate new asset files only if they match all asset file variations. Set the new
unionto restore prior behavior.
✨ Both Alpine and Ubuntu Docker images are now available.
- Alpine images are less than half the size of the Ubuntu-based images
ffmpegpackage supports more video codecs
- Ubuntu has better GPU acceleration support
psnet:asset file URIs now support
✨/📦 Node.js version 18.x LTS (long term support) is now supported on all platforms.
Note that PhotoStructure now requires at least Node.js v14 (the oldest currently-supported version of Node.js).
✨/🐛 Prior versions on Windows and Raspberry Pi on slow disk could result in invalid file lock timeouts, which could prevent some file types (like large HEIFs) from being imported. This could show up as
ENOENTerrors in your sync report or logs.
✨/🐛 Some camera models (like the Galaxy S8+) can produce images that have JPEG encoding errors. Prior builds would prevent importing of these images. (Thanks for the example images, @nighthawk!)
Handling these images required a couple changes:
A new setting,
imageFailOn, lets you import images that have minor encoding defects by default, but still reject images that have been truncated.
Default values for the image validation patterns,
validationErrorAllowlist, now handle more corruption patterns.
processPrioritysetting no longer supports
AboveNormal. If your settings used this value,
processPrioritywill resort to the default,
AboveNormalonly worked if PhotoStructure was running as root (which it never should do!)
/site.webmanifestfile is now dynamically generated, and includes a proper
start_url(so every launch will pick a new seed) and defaults to
Image hash comparison information, including all correlations, deltas, and thresholds, are now included for files (which may help tune
Dominiant colors now include friendly names and percent image coverage.
Limit output to only image hash metadata with the new
pathToLibraryAssetis now rendered for every file to help debug the
When given more than one file applies clustering on the entire array and will return the files provided to ARGV, grouped by asset.
Several additional switches were added to
infoto help customer support, including
✨/🐛 Files with extensions that don’t match their mimetype (say, JPEG-encoded images named
image.dng, which Google Takeouts likes to do) are now imported gracefully.
.psenvfiles are now read correctly when BOM-encoded as UTF-8 or UTF16-LE.
🐛 Glob exclusion patterns were not being applied correctly on Windows
🐛 Newer linux distributions could pull in a version of
heif-convertthat has a buggy filename parser. PhotoStructure invokes this tool in such a way that this bug is avoided.
🐛 Prior builds would cache the absence of
heif-convertuntil restart, which caused confusion for some users. PhotoStructure will now detect newly-installed
heif-convertbinaries within a minute.
🐛 Fixed docker
:stabletagging (see the simplified example)
🐛 Rewrote how tools (like
jpegtran) are detected on the system. Rather than spawning
which, or asking PowerShell for the binary path, we now walk
$PATHlooking for binaries with
$PATHis somehow truncated or invalid, we also walk some default paths (like
%SYSTEMROOT%on Windows, and
/usr/local/binon macOS and Linux).
sqlite.exe not foundand
jpegtran.exe not founderrors on Windows, and should fix SQLite backups on Windows.
🐛 Fixed o.toLocal is not a function, caused if an asset file fails to extact a captured-at time. (Thanks for reporting, @pmocek!)
AssetFileconstraint violation during the asset file cleanup in rebuilds. This could prevent library rebuilds from completing successfully.
🐛 If an Apache reverse proxy closed the SSE socket, PhotoStructure would pop up a “🌩 Not connected” error. This build skips showing that error and tries to quietly restore the SSE socket when broken. See this forum post for details.
🐛 PhotoStructure can now allow drives to go to sleep. It should “just work,” but to set
mountpointsTtlMs=0to force this behavior on platforms that don’t have mountpoint-change-watcher functionality.
mountpointsTtlMsdefaults to 0 on docker now, btw.
🐛 Fixed off-center home icon on Safari (the
displayPathfor the root tag was
[ null ], oops)
🐛 Note: file picker dialogs on PhotoStructure for Desktops that use Linux Gnome can pop-under as a “feature” of Gnome. See https://github.com/electron/electron/issues/32857 for details.
✨/📦 System load is now exposed in the about page and the
✨/📦 File I/O was reduced a bit–permission checks now directly use the
Statsobject if cached, rather than requiring a separate
✨/📦 “Actual path” resolution on case-insensitive systems now ensure the correctly-cased pathname is used for URIs. Incorrect case could prevent cross-platform asset file correlation.
📦 System load on macOS and Linux now average together both proc/cpu metrics as well as loadavg(), which should take into account
📦 Control SQLite’s synchronous mode via the new
📦 If file copies are problematic (you’d see
warnlog entries to this effect), you can now force PhotoStructure to use
cp -af(on macOS and Linux) or
Copy-Item(on Windows) by setting the new
📦 Control PowerShell child process concurrency via the new
📦 Added support for newer NEF and old KDC image formats
📦 New setting
twoDigitCutoffYear: sets the cutoff year after which a string encoding a year as two digits is interpreted to occur in the current century. As an example, a value of “50” would make “49” be interpreted as 1949, and “50” as 2050. See https://moment.github.io/luxon/api-docs/index.html#settingstwodigitcutoffyear for details. This defaults to 3 years in the future (modulus 100) and is updated automatically.
📦 Some string handling previously used now-deprecated
.substr(). PhotoStructure now uses locale-aware grapheme splitting where available, which should prevent high-unicode text from being corrupted.
📦 Volume UUID files, .JSON files, and .TOML files all now support UTF-8, UTF-8-with-BOM, and UTF-16LE-with-BOM encodings
📦 Added 20 new serial-to-model-name translations for recently released smartphones and cameras
📦 Pulled in latest versions of Electron, sharp, node, TypeScript, ExifTool, and other third-party libraries.
📦 When spinning up the
photostructure/serverdocker image, better error messages are now emitted when
📦 Mountpoint and volume extraction is now more configurable. See the new
isExcludedMountpoint()now debug-logs why a given mountpoint is excluded, to help tune these settings.
renice-ing and management should be more efficient, as PhotoStructure now defaults to libuv and only resorts to external tooling on failure.
📦 Supported file extensions and mimetypes are now defined in a single dictionary to ensure they are kept in synchronization. Due to the prior design, several more obscure file extensions (like
.KDC) weren’t handled properly.
sync(it wasn’t used, and was an unnecessary third-party dependency)
📦 Most URLs in text files and emitted to stdout were wrapped in angle brackets, but some apps would interpret the trailing
>was part of the URL (looking at you, UnRaid terminal), which 404ed. All wrapped URLs are just plaintext, separated with whitespace.