2024 PhotoStructure release notes

February 23, 2024

Note that these are notes for versions released in 2024.

Please see the current release notes.

v2024.3.3-beta

-prealpha released 2024-03-20 and promoted to -beta on 2024-03-24

Replaced the ExifTool health check timeout with statTimeoutMs (which defaults to 30s – prior builds timed out after 7 seconds)
🐛 Node.js v21 support: fixed DeprecationWarning: The ``punycode`` module is deprecated.
📦 Update Docker image to Node.js 20.11
📦 Update SQLite tooling to 3.45

v2024.3.2-beta

-prealpha released 2024-03-14 🥧 and promoted to -beta on 2024-03-20

🐛 The warning message Error: env(): failed to read .env file caused sync to fail to run. This warning is now only emitted by the main service, and only if the file exists.
🐛 Database migrations were edited to try to gracefully recover from some types of partially-applied migrations. This should remedy many issues like this.
🐛 Geolocation fields are now deleted if GPS is (0,0). Upgraded libraries will auto-resync all assets tagged with Where|Ghana|Western|Takoradi (the nearest city to Null Island).
📦 The /settings page redirected to the health check page if settings took longer than a second to fetch(!!).
📦 Added several more /System/Volume exclude globs to avoid macOS system subdirectories (thanks for the assist, AlanH!
📦 Add tagGeoSynonyms setting:
Due to EXIF and XMP specification drift, there are several ways for geolocation information to be encoded in files. When PhotoStructure applies the “tagGeoTemplate”, we’ll use these “synonyms” to build the geo tag (first synonym with a value wins). See https://exiftool.org/forum/index.php?topic=13811.msg74413#msg74413 for details.
📦 Add writeGeolocationTagsToLibraryCopies setting:
When enabled, inferred geolocation tags will backfill into Country/State/City tags in the library copy (by default, into an XMP sidecar).
This defaults to false, as reverse-geo lookup results can change over time (and should be done on-demand, rather than stored statically and drift into inaccuracies).
📦 Merged commandTimeoutMs and statTimeoutMs settings–they both defaulted to 30s, and given the presence of taskTimeoutMs (which defaults to 2m) having all three timeouts was confusing.
📦 syncCronTZ now defaults to TZ if TZ is a valid IANA time zone (like “America/Los_Angeles”).
📦 info improvements: include captured-at raw EXIF values for files, and if --load-library is specified, include db library setup metadata (like libraryDbFile, libraryDbBackupDir, and useReplica).

v2024.3.1-prealpha “Zep”

Released 2024-03-08

🐛 Extended database migration timeouts to 2 minutes by default. See dbMaintenanceTimeoutMs setting for details. Should resolve this issue.
🐛 Added new migration to re-assert the Progress table schema. Should resolve this issue.
🐛 The webservice now re-writes settings.toml files from prior versions, to ensure the latest settings are visible. Thanks for reporting, @tkohhh! Older versions of settings.toml are now moved to ./archive (it had been ./old).
🐛 sync won’t be started if any health checks post fatal errors.
🐛 main renders service startup errors to stderr now and still tries to spin up the web service (in an effort to try to get the health check page to the user)

v2024.3.0-prealpha

Released 2024-03-08

✒️ Version format change

I’m adopting a simpler version format: $year.$month.$build, where $build starts at zero at the beginning of the month, and gets incremented for every prealpha, alpha, beta, or stable release. For non-stable releases, -$channel is appended to the version format.

As an example, a build might be v2024.1.7-beta. If it proves sufficiently stable, the same code may be re-released as v2024.1.7.

🗺️ New geo location tagger

PhotoStructure now adds Where/Country/Region/City tags for those photos and videos with Latitude and Longitude metadata.

Note that this feature uses an embedded geo database, so no network access is required. This initial implementation only includes cities with a population of 1000 or greater. See the new tagGeo and tagGeoTemplate settings for more details.

🔃 Sync improvements

Previous builds of PhotoStructure had two work queues: one single-threaded work queue for videos, and one multithreaded work queue for images. This was a ~~hack~~ workaround to prevent concurrent ffmpeg invocations. ffmpeg attempts to use all cores by default, resulting in CPU overscheduling.

We’ve since found a fairly reliable way to single-thread ffmpeg, so sync now schedules both video and image work in a single queue, which greatly simplifies the code, and results in higher parallelism (!!). Anecdotally, prior builds would sync several hundred exemplar videos and photos in roughly 3 minutes. This build now completes that same task in under 90 seconds on the same hardware.

PhotoStructure’s task queuing system was also rewritten. Previous builds used a completely separate SQLite schema and database for work scheduling, in an attempt to keep that workload partitioned from the web service. With the new taskListCap setting, task schedulers receive backpressure if the Task table is “full.” This backpressure ensures the table doesn’t grow unbounded, so it felt safe to migrate it into the models database and delete the work queue database. This also allows web to schedule work for sync reliably without socket RPC or JSON watchfile overhead (again, allowing another good chunk of code to be deleted).

🐕 New stuck-task watchdog

For larger libraries with tens of thousands of ffmpeg transcodes, a sync could get “stuck” waiting for an ffmpeg transcode completion that exited abnormally. v2 builds had a hard timeout value based on video duration, but that proved problematic for slower computers and for more advanced codecs that require more computation to decode, so video transcode timeouts were dropped in v2023. More advanced video transcode timeouts were built that adjusted dynamically based on current system performance and processed pixel count, but this implementation was difficult to test rigorously, and the least-squares interpolation implementation was replaced with a new stuck-task watchdog.

When users reported their sync process was “stuck,” they’d always report that their system’s CPU was idle but that things weren’t done.

So, instead of fancy-pants pixels-processed-per-mimetype least-squares timeout interpolation complexity, why can’t PhotoStructure just do what the users are doing in these situations?

So now it does!

While sync is currently processing, every five minutes it will check if the system load is “idle” (by default, less than 50% of one busy core, but this is adjustable). If it is, any task that has run longer than the last check will be assumed to be stuck, and will be marked as failed. See the new stuckCheckIntervalMs and minBusyPct settings for more details.

To make this work, Tasks are now be abortable externally, and know how to clean up gracefully, including killing child processes and notifying sync-report.

✨ Improvements and bug fixes

✨ Sync is now scheduled by a crontab entry. Prior builds waited a static amount of time between completion of last sync and start of next sync, which resulted in unpredictable sync run times. By default PhotoStructure will now kick off sync every night at 2AM local time, but this is configurable now–and don’t worry, any scheduling overruns are automatically skipped. See syncCron and syncCronTZ settings for details – be sure to set syncCronTZ to ensure “2AM” really is in local time.
🐛 Fixed Error: cannot store REAL value in INTEGER column Progress.completePct. This could cause library upgrades from v1.1.0 to fail as well. Thanks for reporting, Alan!
🐛 Using force-sync via the nav menu in some situations would only work once, as the persistent operation wasn’t resolved after sync completed if there were any rejected tasks. This should be resolved.
🐛 .NoMedia could be ignored in some situations after initial directory scans. This should be resolved.
🐛 There were several edge cases that could prevent sync from properly no-op’ing unchanged files, which could result in sync taking a long time to process previously-scanned directories. This should be resolved.
🐛 PhotoStructure for Docker’s About > Sync Information table could show the library path twice. This should be resolved. Thanks for the report, Gijsh!
🐛/📦 Overlapping “Empty trash” and “Remove assets” actions could result in only a subset of assets actually being removed or excluded. These operations have been converted to the new task infrastructure with serial mutexes to avoid issues around concurrency.
🐛/📦 LibRAW’s support for current flagship mirrorless camera RAW file formats is lacking. PhotoStructure can still show a preview for those RAW images (most of which embed a full-resolution JPEG), so we’re changing the default setting for validateRawImages to false in this build. Future builds will probably switch to rawtherapee for RAW rasterization.
📦 Empty trash and Remove assets now write sync report records.
📦 SyncDirectory writes both scan-complete and sync-complete sync report records with elapsed time.
📦 Added excludeHidden setting:
PhotoStructure may check for filesystem “hidden” metadata flags on macOS and Windows filesystems, and automatically skip importing those files.
As of v2023, this defaults to “false”, as most people don’t use this filesystem feature, and it’s expensive for PhotoStructure to check for this flag on every file it imports.
This setting is ignored on Linux systems.
📦 Added skipWriteVolumeUuidFilesWithNoMedia setting:
When true, PhotoStructure will NOT write files with universally unique identifiers into the root directory of volumes that have been marked with a NoMedia file or folder. If writeVolumeUuidFiles is false this setting is ignored.
📦 Added workQueueHighWater/taskListCap setting:
When PhotoStructure scans a directory, the first step is to walk the directory and search for files to import. When the work queue is larger than this value, sync will pause looking for additional files to process. This limits the size of the work queue to not fall over when there are hundreds of thousands or millions of files to import due to IOWAIT or memory oversubscription. Note that until the last batch of work is scheduled, ETAs will be inaccurate.
Set this to 0 to disable.
📦 Added maxValidFutureMs setting:
If PhotoStructure encounters a year that is more than this value in the future, it will consider that source to be invalid and look elsewhere for the captured-at date for that given file.
Set to 0 to disable future date filtering.
📦 Added forceFilters setting:
When set, all files filters will be applied to visited files. If this is false, files already in the library database will be assumed to be validly passing all import filters. This is set to true by default when rebuilding libraries.
This setting is transient and only set via environment variables.
📦 Setting cpuBusyPercent=0 now supports “single threaded” mode, which tells sync to:
- ignore system load
- consume one CPU core (roughly)
Note that we don’t do CPU pinning, so load from the single-threaded process will probably bounce across portions of different cores, depending on your OS. Expect system load to be about 0.75-1.5 (or about ~75-150% of a core) due to graphics, SQLite, ExifTool overhead.
📦 Database maintenance tasks and relevant health check timeouts have been extended to 1 minute and can be configured with dbMaintenanceTimeoutMs setting, which default to 1 minute per database operation
📦 Maintenance tasks check periodically if the service is ending, rather than running to completion (which could take several minutes, causing the library database to be left in a corrupt state).
📦 taskTimeoutMs and commandTimeoutMs can now be validly 0: previous builds would pass this value directly on to batch-cluster, which does not accept values of less than 10.
📦 AbortErrors are no longer considered “retriable” (which could cause issues under high concurrency).
📦 Tag asset count rebuilds now run non-recursive updates for leaf tags, which are dramatically faster than CTE queries. This can speed up tag asset count rebuilds for larger libraries by more than 5-10x. You can force a Tag.assetCount rebuild with ./photostructure info --reindex.
📦 Replace axios with direct node:http (less dependencies are always better)
📦 Volume and mountpoint watches are now only enabled if scanAllDrives is enabled. This may reduce idle CPU and disk activity.
📦 When available, volume and mountpoint metadata reads directly from /proc/ instead of forking df and mount to gather the same information.
📦 Library rebuilds now serialize only asset re-aggregation tasks. All other steps are parallelized.
📦 UV_THREADPOOL_SIZE can be overridden via the new webUvThreads and syncUvThreads settings:
Higher values may allow for more concurrent requests, but may also consume more memory and CPU and overwhelm non-SSD storage. The default is 4, which should be fine for most installations. Read more about UV_THREADPOOL_SIZE: https://nodejs.org/api/cli.html#cli_uv_threadpool_size_size
🐛 Several tools didn’t respond correctly to the --no-color option.
📦 Non-retriable library database errors now force-close and reopen the database handle, improving error recovery.
📦 Most file operations now use “work-in-progress” files. These files are now unique per-call (using .WIP-${RANDOM_SHORT_UID}-${destination_basename}), which helps avoid issues from inadvertent concurrent file operations.
📦 Database models are now batch-reloaded on upsert, which can dramatically reduce db query load during sync.
📦 When sync is killed or shut down while actively importing files, those files are now marked as a new canceled state in the sync report. The next time sync is restarted, those tasks should be retried automatically. Prior builds would mark those files as failed and require another full sync run to recover gracefully.

🔦 `list` improvements:

📦 The list tool has a bunch of new options:

--primary        Only include the primary, or "best" asset file variation
                 found for every asset. See https://phstr.com/dedupe for
                 details.

--no-primary     Exclude primary asset file variation for every asset.
                 This is mutually exclusive with the --primary option,
                 and returns all rows that option omits.

-0, --print0     Print each full native path name to standard output,
                 followed by a null character (instead of the newline
                 character).
                 This is suitable for properly handling filenames that
                 include whitespace characters in shell pipelines using
                 commands like xargs, which has a "--null" mode which
                 expects filenames to be separated by the null character.
                 This cannot be used with --json or --dump.

--todo           List the currently enqueued files that sync is going to
                 process next. Implies --json. Does not support --print0.

--tags           List all tag paths along with their counts. Implies
                 --json. Does not support --print0.

🐛 list --limit works now. Prior builds could miss adding the sql LIMIT clause to the query.
🐛 list --json now emits a valid JSON array of objects, so you can pipe the output to, say, jq . Prior builds would emit individual JSON objects separated by newlines, which most JSON-consuming tools don’t know how to deal with.

Prior release notes

Release notes