2023 PhotoStructure release notes
This is a detailed list of changes in each version.
-
Major releases have posts summarizing bigger changes. See the posts tagged with “release notes”.
-
New releases, starting in 2023, use “calendar versioning,” or CalVer, using scheme
YY.MM.MINOR
. -
Stable, released versions are recommended. See the forum post for details about alpha, beta, and stable releases.
-
“Pre-release” builds (those that include
alpha
orbeta
) have not been thoroughly tested, and may not even launch. -
Only run
alpha
orbeta
builds if you have recent backups. -
If you update to an alpha or beta build and you want to downgrade to a prior version, know that older versions of PhotoStructure may not be able to open libraries created by newer versions of PhotoStructure. You will probably need to restore your library from a database backup.
23.5.0-prealpha.1 #
(to be released)
(This version’s contents had previously been listed as v2.1.0-alpha.8
, but we’re switching to CalVer, using scheme YY.MM.MINOR
.)
✨ PhotoStructure no longer “fails fast.” #
What’s that mean?
PhotoStructure will try to always stay up and running, even if your library isn’t available, or something’s amiss, like a misconfiguration or something broken on the system.
If anything prevents your library from being open, PhotoStructure will automatically redirect to a new /health
page that list several handfuls of health checks to help people diagnose what’s amiss, and in some cases, buttons that can attempt to repair what’s wrong.

PhotoStructure’s new health check page
This means people running PhotoStructure for Docker without reading the instructions will be presented with a friendly screen with direct links to the relevant documentation.
This change also meant I could put the /about
page on a diet–it only holds fairly cheap and cached content now, so it shouldn’t disconcertinly hang anymore (prior versions ran several “health checks” that were run synchronously whenever the about page was requested).
This change also means PhotoStructure stays up even if your library hard drive gets periodically disconnected.
Read more about this change on the forum.
✨ SQLite improvements #
-
PhotoStructure now automatically figures out the best value for PS_FORCE_LOCAL_DB_REPLICA. Previously, we simply defaulted all docker installs to use a local db replica, whose implementation was problematic with prior versions. This determination is also run within a filesystem advisory lock, to prevent concurrent db setup collisions.
-
Database backups are now always taken “hot.” Prior versions required acquiring a halt-the-world mutex to prevent cold backups from causing SQLite corruption (and prior versions had some codepaths that didn’t acquire the lock durably). Prior versions could also miss copying over the
-wal
write-ahead log, which could also cause SQLite corruption. -
The new db health check now validates file integrity, foreign keys, and that the schema comprehensively matches expectations for the current version.
✨ PhotoStructure for Desktops improvements #
-
The main window now preserves placement (even across screens) and dimensions between runs.
-
The
View
menu now has links to go back, open the log and sync reports directories, the systemsettings.toml
, and the librarysettings.toml
✨ PhotoStructure for Servers improvements #
-
Docker and node editions now have a splash screen to see wth is going on at startup (without having to tail logs). (This page only shows if your library is quite large, your computer is quite slow, or a combination of both).
-
If PhotoStructure can’t open the current library, instead of crashing, a new “PhotoStructure Status” page will be shown with diagnostics to help debug what went wrong and links to how to fix it. This should be a lot more friendly (especially as a first impression) for most people.
-
PhotoStructure’s binaries and supporting files were moved from
/ps/app
to/opt/photostructure
. This move shouldn’t impact anyone, and was made to avoid people being confused by mounting anything to/ps
and hiding the entire installation.
✨/🐛/🏗️ Image deduplication improvements #
-
Dominant color extraction now uses adaptive greyscale prefiltering, iterative k-means clustering, and returns percent coverage per color. This change required a database migration (that will be applied automatically) and a library rebuild (that will be scheduled automatically). See the new
dominantColorPixels
,dominantColorKmeansRuns
,dominantColorMergeThreshold
, anddominantColorGreyThreshold
settings. -
Prior builds relied on a single (mean) image hash algorithm. This build adds two additional, novel CIELAB-based approaches (gradient diff and DCT). Having 3 different hashes dramatically helps both precision and recall.
-
Image hashes now use higher-quality resizing interpolation.
-
New settings to control correlation thresholds:
minImageCoeffPct
,minColorCoeffPct
,imageHashFuzzyDateDelta
,imageHashFuzzyDateDelta
,imageHashRotationDelta
,imageHashDifferentMimetypesDelta
, andimageHashGreyscaleDelta
. -
When comparing two files, if either one of the files has an “imprecise” or “fuzzy” captured-at value (if the source is from the filename, inferred or from
Stats
), the image hash is always used, and the captured-at can be different. Disable this behavior by settingstrictDeduping=true
. Prior builds would skip the image hash comparison in some cases. This change will require libraries to be rebuilt when upgraded to this build.
✨/🐛 Time parsing improvements #
-
Video files are notoriously hard to get correct captured-at timezone offset values. Videos regularly encode the
CreateDate
tag in UTC (even when the file wasn’t captured in UTC!). This results in videos from prior versions of PhotoStructure being wrong by several hours. PhotoStructure now tries to “repair” the UTC timezone into the correct timezone by using either GPS metadata or a timezone offset inferrable via the filename. PhotoStructure also now has prioritized tag extraction: see thecapturedAtTags
and newcapturedAtTagsFallback
settings. Note that some Quicktime tags are not reliably stored as UTC, so we look for more reliable tags before resorting to these problematic tags. If you find a video doesn’t have the correct time in your PhotoStructure library, please email us an example. -
Timezone parsing has been improved to support both IANA and ISO offset formats (both of which have been found in the wild 😠).
-
PhotoStructure handles missing timezones and differing timestamp precision more intelligently now: see
minCapturedAtPrecision
andfuzzyDatePrecisionCoeff
for explanations. -
Google Takeout JSON sidecar timestamps no longer (incorrectly) inherit the current system timezone.
Other improvements and bugfixes #
-
✨ Asset file aggregation is now stricter. Previous versions of PhotoStructure attached file variations to existing assets as long as they matched any asset file associated to the asset. PhotoStructure will now aggregate new asset files only if they match all asset file variations. Set the new
assetAggregation
setting tounion
to restore prior behavior. -
✨ Both Alpine and Ubuntu Docker images are now available.
- Alpine images are less than half the size of the Ubuntu-based images
- Ubuntu’s
ffmpeg
package supports more video codecs - Ubuntu has better GPU acceleration support
-
✨
psnet:
asset file URIs now supportsshfs
-mounted partitions -
✨/📦 Node.js version 18.x LTS (long term support) is now supported on all platforms.
Note that PhotoStructure now requires at least Node.js v14 (the oldest currently-supported version of Node.js).
-
✨/🐛 Prior versions on Windows and Raspberry Pi on slow disk could result in invalid file lock timeouts, which could prevent some file types (like large HEIFs) from being imported. This could show up as
EBUSY
orENOENT
errors in your sync report or logs. -
✨/🐛 Some camera models (like the Galaxy S8+) can produce images that have JPEG encoding errors. Prior builds would prevent importing of these images. (Thanks for the example images, @nighthawk!)
Handling these images required a couple changes:
-
A new setting,
imageFailOn
, lets you import images that have minor encoding defects by default, but still reject images that have been truncated. -
Default values for the image validation patterns,
validationErrorBlocklist
andvalidationErrorAllowlist
, now handle more corruption patterns.
-
-
💔/🐛 The
processPriority
setting no longer supportsAboveNormal
. If your settings used this value,processPriority
will resort to the default,Idle
. (AboveNormal
only worked if PhotoStructure was running as root (which it never should do!) -
✨/📦 The
/site.webmanifest
file is now dynamically generated, and includes a properstart_url
(so every launch will pick a new seed) and defaults todisplay: fullscreen
. -
✨/📦
info
tool improvements:-
Image hash comparison information, including all correlations, deltas, and thresholds, are now included for files (which may help tune
imageHash*Delta
settings values). -
Dominiant colors now include friendly names and percent image coverage.
-
Limit output to only image hash metadata with the new
--image-hash
filter. -
pathToLibraryAsset
is now rendered for every file to help debug theassetPathnameFormat
setting. -
When given more than one file applies clustering on the entire array and will return the files provided to ARGV, grouped by asset.
-
Several additional switches were added to
info
to help customer support, including--read-settings
,--suggested-libraries
, and--child-env
.
-
-
✨/🐛 Files with extensions that don’t match their mimetype (say, JPEG-encoded images named
image.dng
, which Google Takeouts likes to do) are now imported gracefully. -
✨/🐛
settings.toml
and.psenv
files are now read correctly when BOM-encoded as UTF-8 or UTF16-LE. -
🐛 Glob exclusion patterns were not being applied correctly on Windows
-
🐛 Newer linux distributions could pull in a version of
heif-convert
that has a buggy filename parser. PhotoStructure invokes this tool in such a way that this bug is avoided. -
🐛 Prior builds would cache the absence of
heif-convert
until restart, which caused confusion for some users. PhotoStructure will now detect newly-installedheif-convert
binaries within a minute. -
🐛 Fixed docker
:alpha
,:beta
, and:stable
tagging (see the simplified example) -
🐛 Rewrote how tools (like
ffmpeg
,heif-convert
, andjpegtran
) are detected on the system. Rather than spawningwhich
, or asking PowerShell for the binary path, we now walk$PATH
looking for binaries withrx
access. If$PATH
is somehow truncated or invalid, we also walk some default paths (like%SYSTEMROOT%
on Windows, and/usr/bin
and/usr/local/bin
on macOS and Linux).This fixes
sqlite.exe not found
andjpegtran.exe not found
errors on Windows, and should fix SQLite backups on Windows. -
🐛 Fixed o.toLocal is not a function, caused if an asset file fails to extact a captured-at time. (Thanks for reporting, @pmocek!)
-
🐛 Fix
AssetFile
constraint violation during the asset file cleanup in rebuilds. This could prevent library rebuilds from completing successfully. -
🐛 If an Apache reverse proxy closed the SSE socket, PhotoStructure would pop up a “🌩 Not connected” error. This build skips showing that error and tries to quietly restore the SSE socket when broken. See this forum post for details.
-
🐛 PhotoStructure can now allow drives to go to sleep. It should “just work,” but to set
volumeMetadataTtlMs=0
andmountpointsTtlMs=0
to force this behavior on platforms that don’t have mountpoint-change-watcher functionality.mountpointsTtlMs
defaults to 0 on docker now, btw. -
🐛 Fixed off-center home icon on Safari (the
displayPath
for the root tag was[ null ]
, oops) -
🐛 Note: file picker dialogs on PhotoStructure for Desktops that use Linux Gnome can pop-under as a “feature” of Gnome. See https://github.com/electron/electron/issues/32857 for details.
-
✨/📦 System load is now exposed in the about page and the
info
tool. -
✨/📦 File I/O was reduced a bit–permission checks now directly use the
Stats
object if cached, rather than requiring a separateaccess()
I/O call. -
✨/📦 “Actual path” resolution on case-insensitive systems now ensure the correctly-cased pathname is used for URIs. Incorrect case could prevent cross-platform asset file correlation.
-
📦 System load on macOS and Linux now average together both proc/cpu metrics as well as loadavg(), which should take into account
-
📦 Control SQLite’s synchronous mode via the new
dbSynchronousMode
setting -
📦 If file copies are problematic (you’d see
warn
log entries to this effect), you can now force PhotoStructure to usecp -af
(on macOS and Linux) orCopy-Item
(on Windows) by setting the newonlyNativeFileCopy
setting totrue
. -
📦 Control PowerShell child process concurrency via the new
powerShellProcs
setting -
📦 Added support for newer NEF and old KDC image formats
-
📦 New setting
twoDigitCutoffYear
: sets the cutoff year after which a string encoding a year as two digits is interpreted to occur in the current century. As an example, a value of “50” would make “49” be interpreted as 1949, and “50” as 2050. See https://moment.github.io/luxon/api-docs/index.html#settingstwodigitcutoffyear for details. This defaults to 3 years in the future (modulus 100) and is updated automatically. -
📦 Some string handling previously used now-deprecated
.substr()
. PhotoStructure now uses locale-aware grapheme splitting where available, which should prevent high-unicode text from being corrupted. -
📦 Volume UUID files, .JSON files, and .TOML files all now support UTF-8, UTF-8-with-BOM, and UTF-16LE-with-BOM encodings
-
📦 Added 20 new serial-to-model-name translations for recently released smartphones and cameras
-
📦 Pulled in latest versions of Electron, sharp, node, TypeScript, ExifTool, and other third-party libraries.
-
📦 When spinning up the
photostructure/server
docker image, better error messages are now emitted when/ps/library
is missing. -
📦 Mountpoint and volume extraction is now more configurable. See the new
excludedFilesystemTypes
,excludedRootDirectories
andexcludedMountpoints
settings.isExcludedMountpoint()
now debug-logs why a given mountpoint is excluded, to help tune these settings. -
📦 Process
renice
-ing and management should be more efficient, as PhotoStructure now defaults to libuv and only resorts to external tooling on failure. -
📦 Supported file extensions and mimetypes are now defined in a single dictionary to ensure they are kept in synchronization. Due to the prior design, several more obscure file extensions (like
.KDC
) weren’t handled properly. -
📦 Removed
--progress
fromsync
(it wasn’t used, and was an unnecessary third-party dependency) -
📦 Most URLs in text files and emitted to stdout were wrapped in angle brackets, but some apps would interpret the trailing
>
was part of the URL (looking at you, UnRaid terminal), which 404ed. All wrapped URLs are just plaintext, separated with whitespace.