2024 PhotoStructure release notes

February 23, 2024

This page contains a detailed list of changes made in every build of PhotoStructure.

PhotoStructure versions 🔗

Major versions may have posts summarizing more visible changes. Check out the posts tagged with “release notes”.
PhotoStructure uses “calendar versioning,” or CalVer, using the template YYYY.MM.BUILD. BUILD starts at zero at the beginning of the month, so 2024.3.3 is the fourth build from March 2024.

Experimental builds 🔗

Experimental builds end with a -prealpha, -alpha, or -beta suffix.
These experimental builds have not been thoroughly tested, and may not even launch.
Only run these builds if you have recent backups.
If you run one of these builds, please consider hopping into our chat to make sure we can squash any bugs you encounter.

Version upgrades 🔗

Library upgrades to new versions of PhotoStructure are automatic, but older versions of PhotoStructure cannot open libraries from newer versions of PhotoStructure.

Next release 🔗

NOTE: The next several builds will be pre-alpha and Desktop-only 🔗

PhotoStructure for Desktop packaging and upgrading was rewritten, and it’ll take a couple patch releases to get this right.

Hopefully this will only take a handful of days to get all worked out.

Simplified PATH settings 🔗

💔 To simplify configuring external tools, PhotoStructure now uses just the $PATH environment variable and the (optional) PS_TOOLS_PATH setting. Prior builds had a setting for some of the external tools that PhotoStructure relies on (like ffmpegPath, dcraw_emuPath, and heifConvertPath). Read more on discord.

Other improvements 🔗

✨ New dropdown in tag gallery header to the right of the current tag that lists all direct child tags
✨ New +/- in the tag gallery header to control how many rows of thumbs to show
✨ Add “county” support for the geotagger (to help discriminate between same-named cities).

Note that this is accomplished by both a new version of ExifTool as well as a change to the tagGeoTemplate setting. If you prefer the prior geotag format, set tagGeoTemplate=["Country","State","City"] in your library’s settings.toml.
✨ Filename “dash-dash” keyword extraction now uses it’s own keywordDashDashDelimiters setting, which lets you retain whitespace for multi-word keywords (like /photos/--/Places|United Kingdom/P437289.JPG). The previous keywordDelimiters setting is now only used for tags encoded in metadata.
✨ Support for “album.xmp” sidecars: When gathering sidecars for a given photo or video, PhotoStructure will now look for “album.xmp” and “metadata.xmp” files in the same directory, and use the metadata encoded therein as “low priority” sidecars. See the directorySidecars setting for details. In an effort to not impact sync performance too much, the filenames in this setting are case-sensitive, and that these “directory” sidecars are not “inherited” by child directories. Symlinks will be followed, however.
✨ Video metadata is now enriched by ffprobe if installed, which handles (many!) more video types than exiftool and provides per-stream metadata.
✨ HEIF/HEIC thumbnails are now generated by heif-thumbnailer if the command is found, which renders in 10-100ms (heif-convert can take 2-8 seconds, and if we don’t need a full-size render, this dramatically speeds up imports).
✨ ExifTool’s ImageDataHash is now used as an asset aggregation heuristic to ensure assets that only differ in metadata get correctly aggregated together. Note that this feature is best-effort: some RAW and video formats do not support ImageDataHashes yet, but all the prior aggregation heuristics will still apply. See the new imageDataHashType setting for details.
✨ Improve soft-delete support on all platforms, and add proper support on Docker. Read more about docker configuration here.
✨ Added fileSizeEpsilon and mtimeMsEpsilon settings, which control how sync detects changed files.
✨ Added support for Ubuntu 24.04
💔 Dropped support for Node v21.x, as it is now end of life.
🐛 When using the ISO token with assetPathnameFormat, file copies to your library could fail as the filenames would include : (which is an invalid character with many filesystems). The default ISO token is now yyyy-MM-dd'T'HH-mm-ss.SSS, and pathToLibraryAsset() replaces invalid filename characters with “_”. Those characters are “<”, “>”, “:”, ‘"’, “/”, “\”, “|”, “?”, and “*”.
🐛 Fixed The prior version could ignore PS_FORCE_LOCAL_DB_REPLICA in some situations. PhotoStructure now respects this setting again.
🐛 Fix UNIQUE constraint failed: AssetFile. This includes improvements to asset import queries and support for in-place uri upgrades (from file:// schemes to more robust pslib: and psfile: URIs).
🐛 Health checks had a timeout that could cause spurious failures due to incorrect timeout closure boundaries. This should be improved in this build.
🐛 Upgraded to SQLite 3.46.0, which fixes a regression where fts5 indexes could cause integrity checks to fail (which caused several users on Discord to cause sync to wedge!)
🐛 sync now verifies that an Asset is shown before “no-op”-ing any given file.
🐛 Fixed duplicate #id on settings page
📦 Added a new exactFitResolutions setting that, when set, ensures PhotoStructure builds previews that fit within at all of the given resolutions. This should be useful for Chromecast-esque applications.
📦 Added $TZ and, on Docker, $PUID and $PGID to the environment variable settings health check
📦 Added a shut down link from the PhotoStructure health page (so if a library is unhealthy and cannot show the home page, it can be gracefully shut down)
📦 Improved the emoji used for “Share basic installation information?”
📦 defaults.env no longer suggests camelCased versions of transient settings, as transient settings are not supported in settings.toml files.
📦 Updated Docker image to Node.js v20.15
📦 Rebuilt linux-x64 and linux-arm64 tools with Ubuntu 20.04 docker image to fix libc errors with Ubuntu 20
📦 Improved OS & CPU architecture health checks
📦 New tag gallery PRNG now satisfies new random quality checks including dieharder, and is validated against SQLite. Seeds are shorter by default, and the SQL ORDER BY is faster, as well.
📦 renice() will no-op if PS_PROCESS_PRIORITY=NORMAL
📦 PhotoStructure for Node upgrades have been improved.

Upgrades that updated PhotoStructure’s automatic setup steps could crash immediately after an upgrade due to the startup script, start.sh, being written in shell, which doesn’t support on-the-fly file updates.

To avoid this with future upgrades, all of the bootstrap tasks previously done by start.sh are now run by a new bootstrap.js script (which handles upgrades gracefully).

PhotoStructure for Node also included both a ./start.sh and a ./photostructure executable in the root project directory, which could cause confusion for new users: when should “start.sh” be run as opposed to “photostructure”? The “photostructure” file in prior builds was a copy of ./bin/photostructure.js, which complicated the code used to find ./bin/web.js, ./bin/sync.js, and other child process scripts.

In summary:
- The ./start.sh script was replaced with a 2-line shell script that execs ./photostructure. Future documentation will not reference this script, and will only mention the use of ./photostructure
- The ./photostructure file is now a simplified script that simply runs ./bin/bootstrap.js and then ./bin/photostructure.js.
- Running ./start.sh or ./photostructure are now equivalent, and both ensure that PhotoStructure will run properly on your system.
- A new --reinstall argument can be passed to ./photostructure (or ./start.sh) which deletes previously-installed third-party libraries, re-downloads those libraries, and compiles any native libraries that are needed.
📦 PhotoStructure for Node on macOS when homebrew is detected will automatically brew install -q python-setuptools to solve this installation issue.
📦 PhotoStructure’s “.env” support has been simplified:
- To keep start.sh and ./photostructure as simple as possible, they no longer source anything: no conditionals or variable expansion is supported anymore, and you don’t need to export every key/value pair.
- Prior versions of start.sh used if-else conditionals to determine which file to read from, which was confusing to explain and difficult to debug. PhotoStructure now checks /.psenv, $HOME/.psenv, and $PS_ENV_FILE. Any readable file on any of those paths are read. All key/value pairs are last-one-in-wins.
See the documentation for details about .env support.
📦 Normally PhotoStructure will detect and automatically rewrite both system and library settings.toml files to be compatible with the current version. If you would rather PhotoStructure not run this automatic upgrade process, set PS_AUTO_UPGRADE_SETTINGS=false.
📦 Prior versions included System Volume Information/IndexerVolumeGuid in the default for PS_VOLUME_UUID_FILE_PATHS, but that path is only readable by administrators, which we don’t want PhotoStructure to run as, so this path was removed. If you run PhotoStructure with a user with administrative powers and want to restore the prior behavior, copy this into your library’s .photostructure/settings.toml:

volumeUuidFilePaths = [
  ".uuid",
  "System Volume Information/IndexerVolumeGuid"
]

📦 If you’re uncomfortable with PhotoStructure’s default CIE L*a*b color encoding, PS_COLOR_ENCODING=rgboctal is now supported. Note that RGB doesn’t have as good performance for LSH lookups. Note that changing this setting will require a full library rebuild. You can use it temporarily with the info tool to ensure dominant color extraction is working as expected.
📦 Add support for autoUpgradeSettings=false to disable PhotoStructure’s automatic settings.toml upgrade process
📦 Setting either PS_OPT_OUT=true or PS_NO_NETWORK=true will now disable the version healthcheck.
📦 PS_LOG_LEVEL now supports [TOKEN:]LEVEL,... format. For log entries whose context case insensitively contains TOKEN, LEVEL will be used instead of the default log level.

PhotoStructure for Docker improvements 🔗

📦 $PS_LIBRARY_DIR and $PS_CONFIG_DIR are respected when finding defaults for $PUID and $PGID.
📦 $PUID and $PGID are respected when spawning a shell into a new container (not just for starting photostructure main)
📦 $PUID/$PGID library files are chowned, even if the command is not a photostructure subcommand. Set $PS_NO_PUID_CHOWN=1 to skip the chown calls.

v2024.3.3-beta 🔗

-prealpha released 2024-03-20 and promoted to -beta on 2024-03-24

🐛 Replaced the ExifTool health check timeout with statTimeoutMs (which defaults to 30s – prior builds timed out after 7 seconds)
🐛 Node.js v21 support: fixed DeprecationWarning: The ``punycode`` module is deprecated.
📦 Update Docker image to Node.js 20.11
📦 Update SQLite tooling to 3.45

v2024.3.2-beta 🔗

-prealpha released 2024-03-14 🥧 and promoted to -beta on 2024-03-20

🐛 The warning message Error: env(): failed to read .env file caused sync to fail to run. This warning is now only emitted by the main service, and only if the file exists.
🐛 Database migrations were edited to try to gracefully recover from some types of partially-applied migrations. This should remedy many issues like this.
🐛 Geolocation fields are now deleted if GPS is (0,0). Upgraded libraries will auto-resync all assets tagged with Where|Ghana|Western|Takoradi (the nearest city to Null Island).
📦 The /settings page redirected to the health check page if settings took longer than a second to fetch(!!).
📦 Added several more /System/Volume exclude globs to avoid macOS system subdirectories (thanks for the assist, AlanH!
📦 Add tagGeoSynonyms setting:

Due to EXIF and XMP specification drift, there are several ways for geolocation information to be encoded in files. When PhotoStructure applies the “tagGeoTemplate”, we’ll use these “synonyms” to build the geo tag (first synonym with a value wins). See https://exiftool.org/forum/index.php?topic=13811.msg74413#msg74413 for details.
📦 Add writeGeolocationTagsToLibraryCopies setting:

When enabled, inferred geolocation tags will backfill into Country/State/City tags in the library copy (by default, into an XMP sidecar).

This defaults to false, as reverse-geo lookup results can change over time (and should be done on-demand, rather than stored statically and drift into inaccuracies).
📦 Merged commandTimeoutMs and statTimeoutMs settings–they both defaulted to 30s, and given the presence of taskTimeoutMs (which defaults to 2m) having all three timeouts was confusing.
📦 syncCronTZ now defaults to TZ if TZ is a valid IANA time zone (like “America/Los_Angeles”).
📦 info improvements: include captured-at raw EXIF values for files, and if --load-library is specified, include db library setup metadata (like libraryDbFile, libraryDbBackupDir, and useReplica).

v2024.3.1-prealpha “Zep” 🔗

Released 2023-03-08

🐛 Extended database migration timeouts to 2 minutes by default. See dbMaintenanceTimeoutMs setting for details. Should resolve this issue.
🐛 Added new migration to re-assert the Progress table schema. Should resolve this issue.
🐛 The webservice now re-writes settings.toml files from prior versions, to ensure the latest settings are visible. Thanks for reporting, @tkohhh! Older versions of settings.toml are now moved to ./archive (it had been ./old).
🐛 sync won’t be started if any health checks post fatal errors.
🐛 main renders service startup errors to stderr now and still tries to spin up the web service (in an effort to try to get the health check page to the user)

v2024.3.0-prealpha 🔗

Released 2023-03-08

✒️ Version format change 🔗

I’m adopting a simpler version format: $year.$month.$build, where $build starts at zero at the beginning of the month, and gets incremented for every prealpha, alpha, beta, or stable release. For non-stable releases, -$channel is appended to the version format.

As an example, a build might be v2024.1.7-beta. If it proves sufficiently stable, the same code may be re-released as v2024.1.7.

🗺️ New geo location tagger 🔗

PhotoStructure now adds Where/Country/Region/City tags for those photos and videos with Latitude and Longitude metadata.

Note that this feature uses an embedded geo database, so no network access is required. This initial implementation only includes cities with a population of 1000 or greater. See the new tagGeo and tagGeoTemplate settings for more details.

🔃 Sync improvements 🔗

Previous builds of PhotoStructure had two work queues: one single-threaded work queue for videos, and one multithreaded work queue for images. This was a ~~hack~~ workaround to prevent concurrent ffmpeg invocations as ffmpeg attempts to use all cores by default, resulting in CPU overscheduling.

We’ve since found a fairly reliable way to single-thread ffmpeg, so sync now schedules both video and image work in a single queue, which greatly simplifies the code, and results in higher parallelism (!!). Anecdotally, prior builds would sync several hundred exemplar videos and photos in roughly 3 minutes. This build now completes that same task in under 90 seconds on the same hardware.

PhotoStructure’s task queuing system was also rewritten. Previous builds used a completely separate SQLite schema and database for work scheduling, in an attempt to keep that workload partitioned from the web service. With the new taskListCap setting, task schedulers receive backpressure if the Task table is “full.” This backpressure ensures the table doesn’t grow unbounded, so it felt safe to migrate it into the models database and delete the work queue database. This also allows web to schedule work for sync reliably without socket RPC or JSON watchfile overhead (again, allowing another good chunk of code to be deleted).

🐕 New stuck-task watchdog 🔗

For larger libraries with tens of thousands of ffmpeg transcodes, a sync could get “stuck” waiting for an ffmpeg transcode completion that exited abnormally. v2 builds had a hard timeout value based on video duration, but that proved problematic for slower computers and for more advanced codecs that require more computation to decode, so video transcode timeouts were dropped in v2023. More advanced video transcode timeouts were built that adjusted dynamically based on current system performance and processed pixel count, but this implementation was difficult to test rigorously, and the least-squares interpolation implementation was replaced with a new stuck-task watchdog.

When users reported their sync process was “stuck,” they’d always report that their system’s CPU was idle but that things weren’t done.

So, instead of fancy-pants pixels-processed-per-mimetype least-squares timeout interpolation complexity, why can’t PhotoStructure just do what the users are doing in these situations?

So now it does!

While sync is currently processing, every five minutes it will check if the system load is “idle” (by default, less than 50% of one busy core, but this is adjustable). If it is, any task that has run longer than the last check will be assumed to be stuck, and will be marked as failed. See the new stuckCheckIntervalMs and minBusyPct settings for more details.

To make this work, Tasks are now be abortable externally, and know how to clean up gracefully, including killing child processes and notifying sync-report.

Improvements and bug fixes 🔗

✨ Sync is now scheduled by a crontab entry. Prior builds waited a static amount of time between completion of last sync and start of next sync, which resulted in unpredictable sync run times. By default PhotoStructure will now kick off sync every night at 2AM local time, but this is configurable now–and don’t worry, any scheduling overruns are automatically skipped. See syncCron and syncCronTZ settings for details – be sure to set syncCronTZ to ensure “2AM” really is in local time.
🐛 Fixed Error: cannot store REAL value in INTEGER column Progress.completePct. This could cause library upgrades from v1.1.0 to fail as well. Thanks for reporting, Alan!
🐛 Using force-sync via the nav menu in some situations would only work once, as the persistent operation wasn’t resolved after sync completed if there were any rejected tasks. This should be resolved.
🐛 .NoMedia could be ignored in some situations after initial directory scans. This should be resolved.
🐛 There were several edge cases that could prevent sync from properly no-op’ing unchanged files, which could result in sync taking a long time to process previously-scanned directories. This should be resolved.
🐛 PhotoStructure for Docker’s About > Sync Information table could show the library path twice. This should be resolved. Thanks for the report, Gijsh!
🐛/📦 Overlapping “Empty trash” and “Remove assets” actions could result in only a subset of assets actually being removed or excluded. These operations have been converted to the new task infrastructure with serial mutexes to avoid issues around concurrency.
🐛/📦 LibRAW’s support for a number of current flagship mirrorless camera RAW file formats is… not great. PhotoStructure can still show a preview for those RAW images, most of which embed a full-resolution JPEG, so we’re changing the default setting for validateRawImages to be false in this build. Future builds will probably switch to using rawtherapee for RAW rasterization.
📦 Empty trash and Remove assets now write sync report records.
📦 SyncDirectory writes both scan-complete and sync-complete sync report records with elapsed time.
📦 Added excludeHidden setting:

PhotoStructure may check for filesystem “hidden” metadata flags on macOS and Windows filesystems, and automatically skip importing those files.

As of v2023, this defaults to “false”, as most people don’t use this filesystem feature, and it’s expensive for PhotoStructure to check for this flag on every file it imports.

This setting is ignored on Linux systems.
📦 Added skipWriteVolumeUuidFilesWithNoMedia setting:

When true, PhotoStructure will NOT write files with universally unique identifiers into the root directory of volumes that have been marked with a NoMedia file or folder. If writeVolumeUuidFiles is false this setting is ignored.
📦 Added workQueueHighWater/taskListCap setting:

When PhotoStructure scans a directory, the first step is to walk the directory and search for files to import. When the work queue is larger than this value, sync will pause looking for additional files to process. This limits the size of the work queue to not fall over when there are hundreds of thousands or millions of files to import due to IOWAIT or memory oversubscription. Note that until the last batch of work is scheduled, ETAs will be inaccurate.

Set this to 0 to disable.
📦 Added maxValidFutureMs setting:

If PhotoStructure encounters a year that is more than this value in the future, it will consider that source to be invalid and look elsewhere for the captured-at date for that given file.

Set to 0 to disable future date filtering.
📦 Added forceFilters setting:

When set, all files filters will be applied to visited files. If this is false, files already in the library database will be assumed to be validly passing all import filters. This is set to true by default when rebuilding libraries.

This setting is transient and only set via environment variables.
📦 Setting cpuBusyPercent=0 now supports “single threaded” mode, which tells sync to:
- ignore system load, and
- consume one CPU core (roughly)
Note that we don’t do CPU pinning, so load from the single-threaded process will probably bounce across portions of different cores, depending on your OS. Expect system load to be about 0.75-1.5 (or about ~75-150% of a core) due to graphics, SQLite, ExifTool overhead.
📦 Database maintenance tasks and relevant health check timeouts have been extended to 1 minute and can be configured with dbMaintenanceTimeoutMs setting, which default to 1 minute per database operation
📦 Maintenance tasks check periodically if the service is ending, rather than running to completion (which could take several minutes, causing the library database to be left in a corrupt state).
📦 taskTimeoutMs and commandTimeoutMs can now be validly 0: previous builds would pass this value directly on to batch-cluster, which does not accept values of less than 10.
📦 AbortErrors are no longer considered “retriable” (which could cause issues under high concurrency).
📦 Tag asset count rebuilds now run non-recursive updates for leaf tags, which are dramatically faster than CTE queries. This can speed up tag asset count rebuilds for larger libraries by more than 5-10x. You can force a Tag.assetCount rebuild with ./photostructure info --reindex.
📦 Replace axios with direct node:http (less dependencies are always better)
📦 Volume and mountpoint watches are now only enabled if scanAllDrives is enabled. This may reduce idle CPU and disk activity.
📦 When available, volume and mountpoint metadata reads directly from /proc/ instead of forking df and mount to gather the same information.
📦 Library rebuilds now serialize only asset re-aggregation tasks. All other steps are parallelized.
📦 UV_THREADPOOL_SIZE can be overridden via the new webUvThreads and syncUvThreads settings:

Higher values may allow for more concurrent requests, but may also consume more memory and CPU and overwhelm non-SSD storage. The default is 4, which should be fine for most installations. Read more about UV_THREADPOOL_SIZE: https://nodejs.org/api/cli.html#cli_uv_threadpool_size_size
🐛 Several tools didn’t respond correctly to the --no-color option.
📦 Non-retriable library database errors now force-close and reopen the database handle, which should make error recovery more robust.
📦 Most file operations now use “work-in-progress” files. These files are now unique per-call (using .WIP-${RANDOM_SHORT_UID}-${destination_basename}), which helps avoid issues from inadvertent concurrent file operations.
📦 Database models are now batch-reloaded on upsert, which can dramatically reduce db query load during sync.
📦 When sync is killed or shut down while actively importing files, those files are now marked as a new canceled state in the sync report. The next time sync is restarted, those tasks should be retried automatically. Prior builds would mark those files as failed and require another full sync run to recover gracefully.

🔦 `list` improvements: 🔗

📦 The list tool has a bunch of new options:

--primary        Only include the primary, or "best" asset file variation
                 found for every asset. See https://phstr.com/dedupe for
                 details.

--no-primary     Exclude primary asset file variation for every asset.
                 This is mutually exclusive with the --primary option,
                 and returns all rows that option omits.

-0, --print0     Print each full native path name to standard output,
                 followed by a null character (instead of the newline
                 character).
                 This is suitable for properly handling filenames that
                 include whitespace characters in shell pipelines using
                 commands like xargs, which has a "--null" mode which
                 expects filenames to be separated by the null character.
                 This cannot be used with --json or --dump.

--todo           List the currently enqueued files that sync is going to
                 process next. Implies --json. Does not support --print0.

--tags           List all tag paths along with their counts. Implies
                 --json. Does not support --print0.

🐛 list --limit works now. Prior builds could miss adding the sql LIMIT clause to the query.
🐛 list --json now emits a valid JSON array of objects, so you can pipe the output to, say, jq . Prior builds would emit individual JSON objects separated by newlines, which most JSON-consuming tools don’t know how to deal with.