2021 PhotoStructure release notes

2021 PhotoStructure release notes

Note that these are notes for versions released in 2021.

Please see the current release notes.

v2.1.0-alpha.1

This release will happen in 2022. Click here for details.

v2.0.0-beta.1

Released 2021-10-24

  • ✨ On computers with at least 6 CPUs, preview images are now optimized with mozjpeg defaults, which reduces images by up to 30% with minimal reduction in visible quality. See the new jpegMinimized setting for details.

  • 🐛 Sync should now automatically pick up new files. There were a couple issues:

    • v1.0 changed how PhotoStructure keeps track of sync progress, and prior progress metadata would “trick” sync into thinking there was nothing to do for subsequent runs. New sync jobs started after a completed job will start with new metadata, rather than adopting the prior run metadata.

    • v1.0 added readdir caching, and those cached directory contents could prevent new files in existing directories from being picked up. sync now deletes the readdir cache properly after an import completes.

  • 🐛 Fixed unexpected missing asset file id error, which could prevent file variations from being imported properly

  • 🐛 Stale opened-by lockfiles are now properly deleted on startup, and we reduced the default value for the openLockStaleMinutes setting from 1 hour to 5 minutes. This should resolve issues like this.

  • 🐛 For docker users or libraries stored on remote volumes, lock contention could result in the local replica being overwritten with an empty library database. Lock directories are retained much longer now, and replica mtime and db sizes are now used to determine if replicas are “stale.” Local db replicas are now stored in library-uuid-specific subdirectories to prevent overwrites when multiple library are open concurrently.

  • 🐛 Restored title bars that got lost in v2.0.0-alpha.1

  • 🐛 PhotoStructure for Desktop macOS icons don’t fight with the traffic lights any more.

  • 🐛 Fixed undraggable windows

  • 🐛 Monthly vs annual pricing is now handled in the Stripe Checkout page to make initial subscription setup simpler.

  • 📦 SHAs of files that have been determined to be invalid are now added to the ShaBlocklist table, so future sync jobs don’t try to re-import invalid files.

  • 📦 Added info --read-settings /path/to/settings.toml to help debug settings file permission and encoding issues.

  • 📦 Frontend javascript now targets ES2015 to support older devices (like iPads).

v2.0.0-alpha.1

Released 2021-10-05

  • ✨ Initial support for 💖 liking, 🙈 archiving, ⏏️ removing, and 🗑️ deleting assets! plus only

    See the new FAQ page on archive, remove, and delete for details, including instructions on how to enable these features.

  • ✨ Added new “copy path” option in the asset info panel.

  • ✨ (Re)-added a small zoom-on-hover in gallery pages. This can be disabled by setting prefers-reduced-motion in your browser.

  • ✨ Full screen mode! Tap the toggle button in the header or the f key to toggle fullscreen mode.

    A new mode specific to Safari on iPad with a taller header had to be added (it’s pretty!). Note that Safari on iPhones don’t have fullscreen support, but you can get the same effect by navigating to your library, click “share”, pick “Add to Home Screen”, and launch from that icon. See the forum for details.

  • ✨ PhotoStructure learned how to talk! Client and server code can now show informational and error messages (or “toasts” to let you undo some actions, notify you of network problems, tell you when you need to upgrade, and more.

  • ✨ Rebuilt UX view persistence for faster performance through proper component keying and smarter fetch heuristics: clicking the “back” button is instantaneous, and prior scroll positions are correctly retained.

  • ✨ Asset rendering at high resolutions is now faster. Prior builds didn’t trust the database for rotation metadata, which led to a (potentially expensive) exif parsing step before streaming the original asset to the browser.

    If you see any unexpected errors, please do report them in Discord or on the forum!

  • ✨ Search results now include a sticky top banner with asset count (and possibly an “empty trash” or “remove assets” button, depending on the view).

  • ✨/💔 Rotating via the PhotoStructure UI would write Orientation to a sidecar by default. Unfortunately, many applications ignore sidecars. A new sidecarTagBlocklist setting, which defaults to just ["Orientation"], allows most other tags to be written to sidecars (to avoid touching original files), but still write tags (like Orientation) to the image itself.

  • 🐛 Scroll positions in the search and tag views should now be restored properly when clicking the back and forward buttons.

  • 🐛 Tag redirects to non-empty child tags could result in a blank page. The tag fast-forward code wasn’t properly urlencoding tag paths, which caused issues if the leaf tag name included a “/” (like many lens models do!)

  • 🐛/📦 Fixed “neon” video colorspace rendering.

  • 🐛/📦 Tag galleries add a banner to distinguish between child tag samples and direct assets

  • 🐛 If scanPaths was empty, a library health check warning would be raised (“nothing to scan”), which would erroneously pause the sync process.

  • 🐛 Fixed “unexpected missing asset file id”

  • 📦 Background color was changed to 14% white (from 20% white) to accommodate high-gamut displays (both are pretty dark for SDR displays).

  • 🐛/📦 Invalid durations from sidecars are now ignored.

  • 🐛/📦 Progress panel fetching wasn’t paused properly when navigating away from the home panel, which caused unnecessary load on the webserver.

  • 📦 File operations at startup now include the path of the operation (this should de-mystify the “Computing SHA of” messages for large libraries on slow disks).

  • 📦 Header buttons and search input change background color to show focus. Search input autofocuses.

  • 📦 Switched to vue-powered modal dialogs.

  • 📦 Made font sizes across the UI more consistent. I hope you like 14px!

  • 🐛 Dropdown menus now dismiss by clicking anywhere away from the menu (and, critically, that click isn’t passed through to anything underneath): the non-standard dropdown close button was removed as well.

  • 🐛 Fixed "bad startAtNativePath: ignoring" error that could cause sync to mistakenly skip over scan paths.

  • 🐛/📦 Asset rotation code was refactored and should be more reliable: also, if anything goes wrong now, we now show an error explaining what happened.

  • 📦 Upgraded dependencies (including sharp, electron and a new version of node in the docker image). These changes are non-trivial enough to warrant an alpha release-candidate build.

  • 📦 Added --silent to PhotoStructure for Node’s call to npx yarn install, which hides messages like The platform "linux" is incompatible with this module." Thanks for the suggestion, mnaoumov!

  • 📦 PhotoStructure for Node’s ./start.sh now echoes the shutdown messages emitted from main after trapping SIGINT/ctrl-c (prior builds killed the tee which meant you didn’t know why PhotoStructure wasn’t giving you back your prompt).

  • 📦 PhotoStructure for Server process names are now named photostructure main, photostructure sync, … to make them easier to grep for.

  • 📦 Example searches aren’t added to the “prior searches” list

  • 📦 Long tag names in the asset info panel are now truncated (rather than confusingly scrolling to the right)

  • 📦 keywordDelimiters can be set to the empty string, to avoid splitting on commas or semicolons

  • 📦 PhotoStructure for Servers emits the library path as a URL on startup. To open the directory in a finder window on linux, just ctrl-click the URL. On macOS, right-click the URL and pick Open URLcdf.

  • 📦 PhotoStructure for Servers now renders this on startup if --expose isn’t enabled:

PhotoStructure is ready: <http://localhost:1787/>
See <https://photostructure.com/faq/remote-access/> to enable non-localhost access.

Several tool improvements:

  • 📦 Add --is-library switch to info which runs a set of tests to verify that a given library is satisfactory.

  • 📦 Add --flat switch to info to only emit values of objects with only one key (useful when used with --filter to pluck out a specific value for use by another script).

  • 📦 Several tools erroneously included options that weren’t necessary or relevant, and have been removed.

  • 📦 --help verbiage has been improved for all tools, including some URLs to relevant documentation.

v1.1.0

Released 2021-08-15

  • ✨ Want to see if a similar image is already in your library? The new ./photostructure list --looks-like /path/to/image.jpg switch shows existing, imported assets that share the same SHA or image hash as /path/to/image.jpg. See the forum discussion for more details.

  • ✨ Deduplication improvement: JPEG+RAW pairs with different sets of camera serial number tags are now matched properly. Thanks for the report, bdillahu!

  • 🐛/💔 The PhotoStructure for Desktops setting, openAtLogin, was deleted. The “Open at login” menu option is still available on macOS and Windows, though!

    What we wanted was to have the “Open at login” menu option reflect the current state of the system, and if toggled, sets the system to match that state.

    By having this backed by a setting and the system state, though, the persisted setting would overwrite the system setting whenever PhotoStructure launched. The setting was simply unnecessary.

  • 🐛 The about page and info tool no longer crash if the library is missing.

  • 🐛 Squelched ENOENT errors from expected readdir failures.

  • 🐛 Flushed prior caches for directories that have been touched or unlinked.

  • 📦 docker container run photostructure/server ... doesn’t require a /ps/config bind mount to run.

  • 📦 New ffmpegThreads setting supports customizing the value passed to ffmpeg -threads. A new, more conservative default should also help prevent ffmpeg from chowing all your CPU and then some.

  • 📦 Twiddled autoUpdater.quitAndInstall() to use defaults: we’ll see in v1.2.0 if that allows upgrades to re-launch properly.

v1.0.0

Released 2021-08-12

🎉 PhotoStructure is out of beta! 🎉

Read more about version 1.0.0, PhotoStructure’s biggest release ever.

Note that version 1.0.0 includes all changes made in -alpha and -beta pre-releases.

v1.0.0-beta.15

2021-08-11

  • ✨ Added new sync information section to the About page

  • 🐛 Search works again. Thanks for the report, @Sergi!

  • 🐛 Improved back-button navigation on tag and search pages (previous restoration of scroll position between tags didn’t work correctly)

  • 🐛 Replaced rem CSS references with px to make browser rendering more consistent

  • 🐛 Rewrote rendering of paths in the asset info panel.

  • “Auto-collapsing” child tag improvements:

    (When navigating tag galleries, if a tag has only one direct child tag and no direct assets, PhotoStructure will “fast forward” to the child tag to save you clicks/confusion. See the forum post for more context.

    • 🐛 Fixed paging bug.

    • 📦 Added fastForwardEmptyTags setting to support disabling this behavior if you find it confusing.

  • 📦 Added hostname setting to support explicitly setting the hostname (as another workaround to OSes that change hostname arbitrarily, and a way to pass the hostname into docker containers).

  • 🐛 Fixed automatic sync volume scheduling. Thanks for the assist, @tkohhh!

  • 📦 Sync path scheduling heuristics were adjusted to ensure every path will get synced, even if sync takes longer than syncIntervalHours. We now return:

    1. the last-updated sync path, if it was recently worked on, or
    2. the first listed sync path that’s never been started before
    3. the first listed sync path that’s never been completed before
    4. the least recently completed sync path that hasn’t been completed in the last syncIntervalHours

  • 🐛/📦 Filesystem caches are now flushed at the beginning of sync to ensure new directories and files are seen (which should help sync from missing newly-created directories)

  • 📦 Opened-by locks are now matched by hostname or by system UID (which may help docker and macOS, both of which change host names arbitrarily)

  • 📦 PhotoStructure for Desktops now takes a library database backup when the library is on a remote filesystem and the system is going down to suspend or shutdown, unless backupRemoteDbOnSuspend is set to false. This may help macOS which can disconnect remote filesystems during powernaps.

  • 📦 When the system is shutting down or suspending, library database backups now skip CPU or disk intensive tasks (like VACUUM, OPTIMIZE, or db maintenance tasks), which should help ensure the backup completes quickly and successfully.

  • 📦 Searches with no results show a flash message to that effect.

  • 📦 currentLibraryLockOwner and libraryHasSettings caches no longer expire (which should reduce PhotoStructure shutting down due to filesystem hiccups)

v1.0.0-beta.14

2021-08-05

  • 🐛 PhotoStructure could fail to launch on case-insensitive filesystems due to userdir merging code. Thanks for the report, awojtas and bjarte!

  • 🐛 Cleanup jobs (like removing old logfiles, image caches, and readdir caches) are properly scheduled now. Thanks for the bug report, bdillahu!

  • 🐛 opened-by lockfiles from prior versions are automatically cleaned up (to avoid these errors).

  • 🐛 Library write tests were run more often in beta.13, but any network fileshare hiccups could cause PhotoStructure to shut down. We now only run the write test once, at startup, only on the main process. Thanks for the reports, tkohh and awojtas.

  • 🐛 Cross-volume database backups could fail, causing PhotoStructure to shut down after several hours. This was from beta.13 switching from fs-extra.move() to fs.rename(), and was reverted in beta.14.

  • 📦 Is your library stored on a remote volume? The new maxSyncFileJobsWhenRemote (which defaults to 2) should avoid overwhelming the I/O on slower NASes, which can cause sync to crash (especially when PhotoStructure is running on high-core-count servers). See this post for more details. Thanks for the idea, Mike!

  • 📦 Added movflags +faststart to the default ffmpegTranscodeArgs. Thanks for the suggestion, Zirro!

v1.0.0-beta.13

2021-08-01

  • 🐛 Fix sorting to be reverse chronological order for child asset galleries (prior builds had inadvertently changed order of child assets to chronological order). Thanks for the report, mariomare22!

  • 🐛 Prior versions of macOS volume parsing would completely ignore quarantine and nobrowse mountpoints, but this would cause PhotoStructure to have an incomplete and incorrect view of volumes, leading to invalid Asset File URIs.

  • 🐛 Windows upgrades after beta.13 should restart automatically.

  • 🐛 Simplify sync to check every hour for pending volumes to sync.

  • 🐛 Free disk space now uses realpath to ensure symlinks and crossmounts (common in macOS) look at the correct volume.

  • 🐛 Duplicate mountpoint volumes from GVFS and df are now merged

  • 📦 copyToLibraryMimetypes: support for selective automatic organization (say, to only copy raw images into your library). When “automatic organization” is enabled, files whose mimetypes are included in this list will be copied into your originals directory.Note that mimetypes can include an asterisk to do glob-matching. See the related system setting “copyAssetsToLibrary”. Defaults to ["image/*", "video/*"].

  • 📦 matchSidecarsFuzzily: set to false to disable fuzzy sidecar matching. If set to true, PhotoStructure will look for sidecar files that match the basename of the asset, plus some common suffixes (like “-edit”, “-edited”, or variant copies, like “-2”). This setting only impacts .XMP, .MIE, and .EXIF sidecars. PhotoStructure always matches .JSON files fuzzily, as that is required to handle Google Takeouts properly. Defaults to false (to be conservative with sidecar matching).

  • 📦 Suggested library directory improvements

    • Root mountpoints and direct child directories are checked for prior libraries
    • Suggested directories are verified that they should be read/writable
    • info --suggested-libraries exposes these volumes on the command line now.

  • 📦 New siblingInferenceBasenameCoeff setting controls siblingInferenceBasenameCoeff

  • 📦 Volume metadata is now automatically cached for up to a day, based on the amount of free disk space is available. Thanks for the suggestion, Rodger!. Use info --volumes-ttl --debug to see what PhotoStructure thinks.

  • 🐛/📦 volumes() and mountpoints() results are now cached, so if the OS times out or a disk is flaky, the prior successful result will be returned (instead of crashing).

  • 🐛/📦 Timezone improvements

    • check for .tz and ExifDateTime fields in self and siblings
    • prevent the current TZ from “leaking” into unspecified captured-at times
    • respect enableSiblingInference
    • when inferred, the capturedAt.src now contains information about where the timezone came from

  • 📦 New debugTimeouts setting which logs debug information whenever any operation times out. Defaults to false.

v1.0.0-beta.12

2021-07-20

  • ✨ Slow directories (those that don’t readdir in less than readdirCacheSlowMs) are now automatically excluded from tag inference to prevent sync from bogging down on slow/huge directories.

  • ✨ Tag inference can now be completely disabled with the new setting, enableSiblingInference=false.

  • 🐛 beta.10 didn’t properly migrate libraryPath, and for docker users, the prior behavior defaulted writing settings directly into /ps/config, but beta.10/11 wanted to write to /ps/config/PhotoStructure, which resulted in people unexpectedly seeing the welcome page or Error: code ENOENT: ENOENT: no such file or directory, open '/ps/config/PhotoStructure/settings.toml'. The old and new configuration directories will be automatically merged the first time beta.12 runs.

  • 🐛 Lifted droopy traffic lights on macOS

  • 📦 macOS now has /opt/homebrew/bin and /opt/homebrew/sbin added to the toolPaths setting default.

  • 📦 minDiskFreeGb can now be set to 0 to disable free disk space health checks.

  • 📦 Large library databases will automatically increase maxMemoryMb and maxRssMemoryMb values to accommodate larger SQLite caches. If non-default values have been set by the user, though, they will be respected.

  • 📦 Upgraded all outdated dependencies, including SQLite 3.36.0

v1.0.0-beta.11

2021-07-14

v1.0.0-beta.10

2021-07-14

  • ✨ Added validationErrorAllowlist to fix false-positive JPEG validation errors from smartphone photos that aren’t correctly encoded

  • ✨ Long tag hierarchies automatically collapse in tag gallery views.

  • ✨/🐛 Volume metadata is now multithreaded to scale to systems that have many attached drives and very slow network drives. See the forum for details.

  • ✨/🐛 MacOS and Windows users with tons of volumes: UUID extraction is now done in parallel, which should avoid “Failed to scan system volumes” errors.

  • ✨ Added new keyword re-parenting setting, keywordReparenting

  • ✨ Image hashing performance has been doubled (!!) thanks to in-memory binary preview extraction.

  • 🐛/📦 Process handling was rebuilt

    • Asset importing rejection reasons are no longer “errors,” which caused the sync-file process to be recycled (and slow down the import)
    • Health checks are now automatically run, separate from file importing. Unhealthy processes could result in files being skipped in prior builds.
    • Raw image and video importing are now more reliable. Previous implementations would make a go/no-go decision based on the first buffer seen on stderr output–but ffmpeg streams warnings and errors throughout the processing of a video, so this go/no-go was based on incomplete data.

  • 🐛 Added another swing at fixing file access errors on Desktop on macOS by calling scandir against the library, user directories, and scan paths from the main process.

  • 🐛 sync didn’t idle in beta.9.

  • 🐛 startPaused now holds true for new libraries.

  • 🐛 Tag counts were incorrectly cached, which could prevent new libraries from showing tag counts properly.

  • 🐛 Improved docker mountpoint filtering (scans could get “stuck” in /proc/* volumes)

  • 🐛 Remote MacOS AFP volume hostnames are now properly parsed.

  • 🐛/📦 Nav menu improvements on mobile: more reliable scrolling on iOS Safari, and the bottom element should now be visible (even when the bottom button bar is shown)

  • 🐛 Filepath tags don’t show volume SHAs anymore

  • 🐛 File paths in the Asset Info panel could have rendering issues

  • 🐛/📦 macOS Big Sur versions are now rendered properly on m1 machines

  • ✨/📦 The strictDeduping setting now automatically sets 9 deduping settings to “very strict” values.

  • ✨/📦 Root filesystem tags are now configurable. See the new tagDisplayNameFSLabels and tagDisplayNameFSRoot settings.

  • 📦 Fixed the progress caret

  • 📦 Fixed the nav menu pointer (thanks, Cowherd!)

  • 📦 Improved asset deduping when pairs have different captured-at precision

  • 📦 forceRebuildPreviews now re-transcodes videos (handy for benchmarking ffmpegHwaccel)

  • 📦 --no-filter now disables all filters (including NoMedia)

  • 📦 The server dockerfile no longer specifies VOLUMEs

  • 📦 Changed the default for ffmpegHwaccel from auto to disabled. Docs suggested that auto would be safe, but in practice some platforms (like macOS) throw errors. Feel free to try it out on your box, but don’t be surprised when it doesn’t work… 😞 (see this forum post)

  • 📦 The config, library, and cache dir now remove rwX from Group and Other to help with security.

  • 📦 main now runs health checks on the web service every minute, and the sync service every 15 minutes, just to reduce system load.

  • 📦 maxSyncFileJobs and sharpThreadsPerJob can be overridden (if cpuLoadPercent doesn’t do what you want).

  • 📦 .cache directory cleanup is more efficient now (the prior implementation could get “stuck” if concurrent writes happened during sync)

  • ✨/📦 logcat now reads from stdin when no filenames are provided.

  • 📦 The libraryPath/PS_LIBRARY_PATH setting has been renamed libraryDir/PS_LIBRARY_DIR. This matches all other directory settings. An alias was added for backward compatibility.

  • 📦 Added new remoteFilesystemTypes that defaults to sshfs and s3fs (note that non-FUSE filesystems are already handled properly)

v1.0.0-beta.9

2021-06-17

  • ✨/🐛 Fixed a race condition in child process handling affecting videos and raw images that could cause these files to not be imported.

v1.0.0-beta.8

2021-06-17

  • ✨ Added new disableAllFilters setting, that forces all filter settings to their most permissive value.

  • ✨ Added new ffmpegHwaccel setting enabling FFmpeg hardware-accelerated transcoding.

  • ✨ Transcoded videos now support high gamut output.

  • ✨ Added several new date parsers, new extraDateTimeFormats defaults, and CreationTime to capturedAtTags.

  • 🐛 Fixed paging for tags with large numbers of the exact same captured-at time

  • 🚅/🐛 Updated the is-this-file-in-sync test to handle remote filesystem timestamp skew. This should also help speed up re-syncs of existing volumes.

  • 🐛 Full-text search indexes and asset tag counts are now only rebuilt when changed. This should remove the high I/O issue and help with slow rebuilds.

  • 📦 “Send recent logs” was removed from the nav menu (context was frequently insufficient, and new versions of sentry broke the prior implementation)

  • 📦 Upgraded all dependencies, including electron 13.

v1.0.0-beta.7

2021-06-14

v1.0.0-beta.6

2021-06-14

v1.0.0-beta.5

2021-06-08

v1.0.0-beta.4

2021-06-08

  • ✨ Video assets in galleries now show a duration (the video icon wasn’t added in an effort to reduce the number of DOM elements on the page and get Firefox to render faster)

  • ✨ Tag gallery thumbnails now have a captured-at title (hover over photos to see when they were taken).

  • ✨ JSON Takeout sidecars for images ending in -edited or -1 are now paired properly.

  • ✨ Better crash recovery: irreparable SQLite replicas found in the cache directory are now replaced with the primary database automatically.

  • 🚅/📦 Tag asset counts in prior versions were updated monolithically (every tag’s asset count was recomputed every 5 minutes during syncs). This approach is reasonably fast for smaller libraries, but for larger (200k+ asset files) libraries (especially not stored on SSDs), these queries could take > 30 seconds, which had a large performance impact on filesystem synchronization.

    Tag asset counts are now updated incrementally, and the update query itself has been improved as well (even with very large libraries, incremental tag counts take under 1 second with the new code).

  • 🚅/📦 Images on all browsers that aren’t Safari now use lazy loading. This helps improve the page load time on full-screen 4k displays, but Chrome is still instantaneous, and Firefox takes seconds (at least on Linux).

  • ✨/📦 De-duplication improvement by adding MetadataDate to the default set of captured-at tags.

  • 🐛 Restored PhotoStructure’s periodic sync job

  • 🐛 Docker license validation errors and should be fixed now. Apologies if you saw this!

  • 🐛 PhotoStructure no longer looks for sidecar metadata for files within .photostucture directories

  • 🐛 Gallery rendering has been improved for mobile displays, which in prior versions sometimes showed blank grey rectangles at the bottom right.

  • 📦 SQLite backups now try to use the backup API, and if that fails, tries the new VACCUM INTO command.

  • 📦 Docker users: A UMASK environment variable, if set, will be given to umask during startup. The default is 0022 which removes “group” and “other” write permissions. You may want a more restrictive 0027 which removes read access to “other”.

v1.0.0-beta.3

2021-05-20

  • ✨ PhotoStructure now imports invalidly-named UUID-like JSON sidecars from Google Takeouts

  • ✨ PhotoStructure for Desktop’s splash screen now renders both the version and startup progress.

  • ✨ Date parsing now uses a customizable set of datetime formats. See the new extraDateTimeFormats setting for more information.

    If you have an old LG phone, you might want to add MM.dd.yyyy HH:mm:ss to this setting (I didn’t/couldn’t add this to the default list, as the format ambiguously defines MDY and DMY, and ambiguous parsing leads to pain and suffering).

  • 🐛 On initial library scan, a “library is empty” message would be rendered due to incorrect asset-count caching. This has been fixed.

  • 🐛 Fixed the “update now…” menu item which could result in breaking the current installation.

  • 🐛/📦 Computers in “dark mode” could obscure parts of the PhotoStructure plus checkout page and the email with the verification code. We now explicitly set the background color to fix this issue.

  • 🐛/📦 Both environment and file settings can now use either PS_SHOUTY_SNAKE_CASE or camelCase names of settings, (like PS_LOG_LEVEL and logLevel) in an effort to make things “just work.” We recommend PS_SHOUTY_SNAKE_CASE for environment variables, though, just to avoid PhotoStructure settings from colliding with any other software.

  • 🐛/📦 The info tool now has a working --filter argument to filter output to only fields you’re interested in

  • 📦 Logfiles now automatically rotate every day

  • 📦 As part of PhotoStructure’s “health checks,” it verifies that a library volume can be written to. These “write tests” are called .tmp-XXXXX/write-test.jpg and live in your originals directory. In some cases these may be created but not successfully deleted. PhotoStructure now tries to clean up prior .tmp-* directories when performing health checks.

  • 📦 PhotoStructure for Desktops now shuts down (rather than running in the background) if the main window is closed and a library hasn’t been set up yet.

  • 📦 PhotoStructure for Desktops fills the screen on 1080p displays. On larger displays, it only takes 80% of the screen.

  • 📦 Upgraded all dependencies, which included updates to sharp, electron, and knex

  • 📦 PhotoStructure for Desktops is now being signed by an extended-verification code signing certificate from Sectigo. Prior releases were signed with a certificate from DigiCert. This should be a transparent change.

v1.0.0-beta.2

2021-05-07

  • 🐛/📦 PhotoStructure for Node’s start.sh now works under bash, Cygwin, and from Git for Windows

  • 📦 Timeouts are now configurable (because what PhotoStructure really needed was MOAR SETTINGS). The defaults are the same as before, but those of you with 10+ external volumes or (very eh, lethargic?) HDDs can now double or triple these timeouts.

v1.0.0-beta.1

2021-05-05

  • ✨ Added a first pass at tag normalization: the “Album”, “Keyword” and “Who” tags now all support “aliases”, controlled by the new rootTagAlbumsAliases, rootTagKeywordsAliases, and rootTagWhoAliases settings. This helps normalize tags in existing files, say, if your keywords aren’t (all) in English, or have inconsistencies (“Keyword/Sky” and “Keywords/Sky”).

  • ✨ Dropdown menu improvements

    • 📦 Dropdown menus are now positioned to ensure the content is visible. If possible, they show to the right and below the click target, but will render to the left or even above the click target if the right or bottom screen edge is close.
    • 📦 Dropdown menus now have a close button.
    • 📦 Dropdown menus can now be dismissed by clicking or tapping anywhere else on the screen, and these taps and clicks that dismiss a menu are ignored.
    • 📦 Dropdown menu contents can scroll vertically if their content is too tall for the screen.

  • ✨/📦 Improved UID/GID handling for those upgrading PhotoStructure for Docker from v0.9.

    Prior to v1.0, PhotoStructure for Docker defaulted to running as root. To prevent upgrades from failing due to permission issues, we now default the userid to match the current owner of /ps/config/settings.toml.

    This default value will be overridden if UID, GID, PUID, or PGID are set, which is recommended to minimize both the number of files owned by root, and the number of processes running as root (even within containers).

    If the file is missing, we default to 1000 for both the UID and GID, which is the default for the first non-system user (at least in Ubuntu and Fedora).

  • 🐛 Fixed PhotoStructure for Desktops packaging: alpha.7 was erroneously omitting required third-party .so/.dlls.

  • 🐛 Name parsing for family names that end in “i” is now fixed.

  • 🐛 The when: and date: search query filters are now synonyms, and normalize to date: (details).

  • 🐛/📦 Tag paths are normalized to “NFC” Unicode Normalization Form now. This should help library portability between macOS and other platforms.

  • 🐛 Asset streams are now always scrolled into view on navigation (this was broken in alpha.7)

  • 🐛 Navigating from the welcome page to the plans page could take a while if the user had many slow drives. An aggressive timeout (3 seconds) was added to prevent the page from hanging.

  • 🐛/📦 The “shown” asset file variant in the asset info panel is easier to select now (the prior click target was only the pathname)

  • 🐛/📦 Searching for before:2022 now works (if you want a query that returns all assets in your library). Previous versions applied the “captured at” validity filter to dates, which reject dates more than several weeks in the future.

  • 📦 The View menubar items were re-ordered

  • 📦 Environment and TOML settings now support “aliases.” This allows us to change or improve the names of settings between versions but not break existing configurations. These aliases are listed per setting in the library and system settings.toml files now.

  • 📦 settings.toml and defaults.env now enumerate all environment and TOML key aliases (so prior setting names can be found).

  • 🐛/📦 Clicking “continue” from /welcome now has an aggressive timeout to prevent hangups during the initial installation flow. If you have tons of slow external disks attached to your computer, this should help.

  • 🐛 Subscription signups could fail sometimes when the Stripe API reported that newly-created customers didn’t exist. We retry these requests automatically now to work around this issue.

  • 📦 The info tool now supports filtering output, including deep fields. See info --help for more information.

  • 📦 PhotoStructure for Node’s start.sh script now

    • 📦 verifies that Node.js is at least v14.16.0
    • 📦 warns if ffmpeg isn’t available
    • 📦 warns Ubuntu 18 users that HEIF isn’t supported

  • 📦 Ubuntu 18 fixes:

    • 🐛 lsblk parsing supports older versions (that don’t support volume size information)

    • 📦 rebuilt binaries on 18.04 (prior alpha builds had binaries built on Ubuntu 20.04)

v1.0.0-alpha.7

Released 2021-04-16

🔍 Search improvements

  • ✨/🐛/📦 Search had been query-as-you-type. Now searches are only executed when the user hits return, clicks the magnifying glass, or taps the “search” key on their mobile keyboard.

  • ✨ Search results are now paged via lazy-loaded infinite scroll

  • ✨ The search button on non-root tags now pre-fills the search icon with the current tag to support paging through all inherited assets and as a start to adding more search criteria.

  • ✨ You can now clear prior searches (there’s an X to the right of the heading)

  • 🐛 Fixed the prior search list

  • 🐛 Query parameters from search no longer transfer when returning to the home page

  • 📦 Reworked the search examples

  • 📦 Search queries with unmatched quotes, AND, or OR are no longer considered valid.

  • 📦 The search button isn’t hidden on mobile devices anymore.

🎯 Other improvements

  • ✨ Keyword extraction is now more configurable

  • 🐛 docker-entrypoint.sh now execs node, which lets docker shutdown propagate to PhotoStructure, and let it shut down gracefully (and avoid “Library is already opened by” errors)

  • 📦 New openLockStaleMinutes setting, which defaults to 1 hour: If an opened-by lockfile has not been touched in this number of minutes, the file is considered stale and invalid. Libraries will refresh their lockfile more frequently than this period. The disk that hosts your library won't be able to idle if this is set too short

  • 📦 yarn.lock is now copied into the docker container to ensure consistent contents

  • 🐛 -alpha.1 changed some paths from PhotoStructure to photostructure. -alpha.2 reverted several of these changes, but logfiles were still being written to photostructure. -alpha.7 fixes this.

  • 📦 Upgraded to Electron v12.0.4 and ExifTool v12.24

v1.0.0-alpha.6

Released 2021-04-12

  • ✨ Added meta headers to support iOS homescreen.

  • ✨ Have scanned images of older photos? A new datesBeforeAreEstimated setting automatically considers all captured-at times before 1999 to be an “estimated” time, which requires files to have a tighter image correlation to be considered a duplicate of an existing asset variant. This addresses issues like this.

PhotoStructure for Desktops:

  • 🐛 Fixed the stripe checkout background

  • 🐛 The stripe.com link in the plans page is now clickable

PhotoStructure for Docker:

  • 🐛 Fixed bogus “PS_LIBRARY_PATH must be set” error in /settings

  • 📦 If you are seeing file permission problems, temporarily set the environment variable PS_FIX_PERMISSIONS=1 and run docker-compose up (without --detach) or docker run -it photostructure/server:alpha. This will run chmod -R $UID:$GID /ps as root from within docker, so make sure UID and GID are set appropriately. This chmod should address any permission issues if you previously ran a PhotoStructure container as the root user.

  • 📦 Set the environment variable UID=0 if you want to run PhotoStructure as the root user within your docker container, as it has in prior versions.

  • 🐛/📦 If either the /ps/tmp or /ps/cache directories are bind-mounted, either will be used for the cache directory. This should solve spurious EACCES errors that some alpha testers saw.

v1.0.0-alpha.4

Released 2021-04-11

  • 🐛 Fix docker ENTRYPOINT and UID/GID handling

  • 📦 Docker detection was massaged: PhotoStructure first looks at the PS_IS_DOCKER environment variable. If it’s set to 0 or 1, it’ll use that value. If PS_IS_DOCKER is not set, and the OS is Linux, and /etc/alpine-release exists, PhotoStructure assumes it’s running within docker. Those of you running Alpine outside of docker, please set PS_IS_DOCKER=0 and please send me an email why you run Alpine.

  • 📦 Removed almost all of the ENV settings in the Dockerfile: PS_LIBRARY_DIR, PS_LOG_DIR, … are now given (the same) default values when PhotoStructure sees that it is running within docker.

  • 📦 If you start up Docker and are missing some bind mounts, PhotoStructure will now error with a link to the bind mount volume descriptions.

v1.0.0-alpha.3

Released 2021-04-11

v1.0.0-alpha.2

Released 2021-03-19

  • 💔 I’ve reverted the UserData directory downcasing that was changed in -alpha.1: I believe I’ve fixed the problem with Electron startup that caused this issue. Sorry for the changes!

  • ✨ Say hello to .cr3 support! LibRaw v0.20.1 is now included in all editions of PhotoStructure, as Ubuntu and Fedora distros are tracking older versions that don’t support .cr3.

  • ✨ The main PhotoStructure for Desktops binary now supports command-line options, so running PhotoStructure-1.0.0-x86_64.AppImage --verbose on a terminal is now a thing.

  • ✨ PhotoStructure for Desktops has a new “Open in browser…” in the View menu and the system tray menu.

v1.0.0-alpha.1

Released 2021-03-14

Please note

  • 💔 PhotoStructure for Desktops on Linux now requires at least Ubuntu 20.04. If you’re on Ubuntu 18.04 or earlier, please switch to the server edition.

    By dropping support for 18.04, which seems to be rarely used (according to Twitter and Reddit polls), we can upgrade to newer versions of Electron and Sharp, picking up several security and performance improvements.

New curators

Note that a library rebuild will be kicked off automatically by this new version, but you get a bunch of new goodies:

  • ✨ Directory hierarchy tags: you can now browse by filesystem! These tags are integrated in the asset info panel as well.

  • ✨ Google Photos albums from Takeouts are now detected and albums are imported as tags.

  • ✨ Google Photos JSON sidecars from Takeouts may contain people and pets. These are now detected and imported as tags.

  • ✨ Adobe Lightroom, DigiKam, Picasa, and Google Photos users, rejoice: PhotoStructure now extracts face tags from these apps and adds them to the new “Who” root tag.

  • ✨ If you’re feeling adventurous, set PS_TAG_NAMES_FORMATTER=family/given. This enables PhotoStructure’s new name parser and lets you navigate by Who/Last/First. Details about this new name parser will be added to the website soon.

More storage flexibility

✨ Want to run your PhotoStructure library from your SSD, but copy your originals to a different folder hierarchy?

The new originalsDir system setting specifies the directory to store original images when copyAssetsToLibrary is enabled. Absolute paths are supported. Relative paths are evaluated from your libraryPath. This setting defaults to “.”, which is the same as your PhotoStructure library directory.

This setting needs to be set appropriately on different computers (it won’t be set automatically!)

If you open your PhotoStructure library on a different computer, and that computer doesn’t have access to the volume with your originals, full-screen zoom won’t work, and non-transcoded videos will not play.

If you have a large library and want to use an SSD, we recommend you set your libraryPath to your SSD, and use this setting to store your originals on a larger volume (rather than using the previous previewsDir setting which fragmented your library).

Improved HEIC support

Due to patent and licensing issues, PhotoStructure does not come with support for .heic. Prior versions of PhotoStructure required complicated recompilation steps to support HEIC, and only to the PhotoStructure for Node edition. This version brings HEIC support to all editions, and is substantially easier to install.

Improved image hashing

  • Prior image hashes were made rotation-invariant by normalizing orientation to the quadrant with the least magnitude. Unfortunately, cameras producing JPEG+RAW pairs using “computational imagery” could change the final image regional brightness to change this quadrant, which results in a false-negative match.

    We tried several other algorithms to find a stable orientation (like most-variant-quadrant) but then realized it took less than a millisecond to “brute force” match by checking hashes against all rotation variants.

    The image hash stored in the database is now oriented based on the rastered image orientation, ignoring metadata rotation values. This means we don’t have to read EXIF tags from the file or sidecars in order to produce the image hash.

  • Color image matching is now controlled by the new minImageCorrPct, minColorCorrPct, and minMeanCorrPct library settings. More details are in those settings’ descriptions.

  • When greyscale images are compared, they now require higher image correlation (customized by the new minImageGreyscaleCorrPct library setting).

  • Rotation normalization now uses this new rotation-invariant image hash implementation. Rotations used to take several seconds due to rotation normalization, and is now essentially instantaneous.

Improvements and bugfixes

  • 🐛/👻 Seeing ghosts? The useEmbeddedPreviews boolean library setting has been replaced with the new embeddedPreviews string[] setting to fix incorrect ghostly rendering of some iPhone photos. The same conversion (from boolean to string[]) was also done to useEmbeddedThumbnails => embeddedThumbnails.

  • 💔/📦 The networking settings group was moved from library settings to system settings. This affects localhost, httpPort, exposeNetworkWithoutAuth, and rpcPort. If you’ve configured these via environment variables, you don’t need to do anything.

  • ✨ PhotoStructure now caches readdir() results. Several beta testers have used software that dumped their entire 50k+ photo library into a single directory, which doesn’t play nicely over a remote filesystem.

    By caching readdir() results (see the readdirCacheSeconds setting), PhotoStructure’s sibling inference and file scanning codepaths should be able to not get “stuck” on these sorts of directories anymore.

  • ✨ For PhotoStructure on Desktops users on macOS or Windows, you can now enable Open at login via the PhotoStructure or File menu bar.

  • ✨ PhotoStructure sync processes can now be canceled mid-flight by sending the process a SIGUSR1 signal. This is handy for users that want to run manual sync jobs.

  • ✨ The “best” asset file variant is now configurable via a new variantSortCriteria library setting. See the forum post for more information.

  • ✨ FFmpeg settings are now configurable via the new ffmpegTranscodeArgs system setting, in case you want to use hardware accelleration.

  • ✨ Normally PhotoStructure ignores any volumes that are “unhealthy” (as reported by the OS). This prevents PhotoStructure from doing I/O against that volume which can cause system instability. A new ignoreUnhealthyVolumes setting, which defaults to true, lets you override this behavior.

  • ✨/🐛 Image validation types can now be configured via the new validationErrorBlocklist library setting.

  • ✨/🐛 PowerShell’s startup arguments are now configurable via powerShellArgs which addresses this issue.

  • ✨/🐛 Sibling files that are used for tag inference must now either share a stat time within a day of the target, or have a parsable-to-timestamp filename whose sibling also parses to an adjacent day (this helps prevents PhotoStructure from looking at spurious siblings).

  • ✨/🐛 Square thumbnail cropping is now only performed once (rather than scaling all sizes in parallel). Prior versions of PhotoStructure could generate different square thumbnails for different sizes, as the crop algorithm would behave differently at different resolutions.

  • ✨/🐛 Prior versions of PhotoStructure would decide if a file needed to be transcoded by examining the MIMEtype of the video.

    This works OK for several formats, but for video container types that can store several different codecs, MIMEtype is not sufficiently comprehensive, and for several users, resulted in videos that played audio by not video.

    Instead, PhotoStructure now looks at the container type, and the audio and video codecs used, to see if the video will play correctly on most evergreen desktop and mobile browsers.

  • ✨/🐛 Some users reported incorrect colors in RAW images. To solve this, PhotoStructure now uses libraw instead of dcraw. libraw is actively developed, handles many more raw image types, and is faster as well.

  • ✨/🐛 Try to prevent “tofu” (missing glyphs rendered as empty squares) by including latin-ext font glyphs as well as several common non-latin system fonts.

  • ✨/🐛 When tags now use the new _displayName field to render the i18n‘ed display version of a month. This avoids having different tag names for the same month because the system locale changed between runs.

  • 🐛 “Open file in folder” on Windows could fail if the path had whitespace.

  • 🐛 PhotoStructure could fail to launch if readdir failed for any root directories of volumes.

  • 🐛 Some docker containers befuddled PhotoStructure’s isDocker() detection. This is now forced to true within the Dockerfile by setting PS_IS_DOCKER.

  • 🐛 Network file shares mounted via IP address versus zeroconf may not have properly resolved URNs, which may result in duplicate URIs generated for the same device. If you find this duplication in your asset info panel, please contact support@photostructure.com and we can help get things sorted.

  • 🐛 Cache directory cleanup now gracefully handles filesystems whose caches aren’t strictly updated, which could result in directories that weren’t cleaned up properly.

  • 🐛 JPEG+RAW image pairs can have different GPS locations! This is due to the GPS location being acquired by different sources (cellular vs A-GPS vs WiFi vs actual GPS satellite telemetry). Prior deduping would mark image pairs that were not strictly equal as different images. Current deduping will consider the location to be equivalent if the distance between the two GPS locations are less than gpsErrorMeters, which defaults to 500 meters.

  • 🐛 Fixed “internal error: Error: no pending currentTask” The issue is from sync-file reporting health status from a periodic timer, but because the parent didn’t ask, it flips out. The fix makes the daemon get quietly recycled when health checks fail.

  • 🐛 Exotic timezones are now handled properly.

  • 📦 PhotoStructure for Desktops on macOS now detects and fails if it is being launched from the DMG (instead of having been installed into the Applications directory).

  • 📦 Double-clicking the system tray icon now opens the home page.

  • 📦 Added new useLibraryPathsToInferDates (which defaults to false) to avoid propagating previously-incorrect date parsing (due to the asset having been placed in the incorrect timestamped folder).

  • 📦 Added useStatToInferDates setting (which defaults to true). Setting this to false will omit assets whose captured-at time cannot be extracted except via the filesystem’s stat record (which is not a reliable source for captured-at, as file transfers and backups frequently don’t retain these values correctly).

  • 📦 Added fuzzyYearParsing (which defaults to false). When enabled, PhotoStructure will use directories starting with a number that looks year-like (four digits, 1826-2020) to infer the captured-at time, if all other date parsers have failed. Note that setting this to true “forces” the fuzzyDateParsing setting to be true as well.

    PhotoStructure first looks for metadata with a date, then looks for an ISO-compliant YMD timestamp in the filename or path, and then, if fuzzyDateParsing or this setting is enabled, a YMD or YM datestamp, and then finally, if this setting is enabled, it looks for a directory that begins with a number that is between 1826-2020.

  • 📦 Added minValidYear (which defaults to 1826, the first year a photograph was captured). If you have paintings or other imagery from before 1826, you’ll want to make this value less than the earliest image in your library.

  • 📦 Filenames with YYYY_MM_dd HH_mm_ss datestamps can now be parsed and used for the captured-at time (used only if metadata is missing).

  • 📦 .thm files are no longer looked at as import candidates.

  • 📦/🛡️ PhotoStructure spawns a number of processes (including exiftool and ffmpeg), and passes through inherited environment variables, mostly to ensure locale and TZ settings are correct. To prevent environment values that contain sensitive information, like API access tokens, from either being logged by PhotoStructure, or from being accessed by external tools, all environment variables whose key matches the new sensitiveEnvRegExp setting will be removed. This defaults to keys that contain the strings SECRET, KEY, PASSWORD, or PASSWD (and aren’t a PhotoStructure environment variable).

    Prior versions only logged environment variables specific to PhotoStructure (like PS_LIBRARY_PATH and PS_LOG_LEVEL), so no prior disclosure (except to spawned ffmpeg and vlc processes) should have occurred.

    (Note to beta users: You may want to consider rotating any keys held in your env, just to be safe.)

  • 📦/🐛 Orphaned tags are now properly vacuumed from the library (prior SQL could quietly fail).

  • 📦/🐛 Tag counting temporary tables are cleaned up properly now.

  • 📦/🐛 The HistoryWhen tag is no longer considered a possible valid value for an asset’s “captured-at” time (as it pertains only to when the History tag was modified).

  • 📦 All health checks can now be individually disabled. This allows disks to spin down when idle, but also means PhotoStructure may not be able to detect and automatically recover from network, file system, and internal glitches. See healthCheckDb, healthCheckExiftool, healthCheckFreeSpace, healthCheckLibraryIsWritable, and healthCheckVolumes.

  • 📦 PhotoStructure for Desktops now detects some common initial system setup issues, and now asks the user if it can open a browser window to the appropriate (hopefully helpful) article on PhotoStructure.com.

  • 📦 For PhotoStructure on Desktops users on Windows or Linux, you can now pick to enable or disable the “tap the alt key to toggle menu bar visibility” via the View menu bar.

  • 📦 To help expedite shutdowns, PhotoStructure now skips maintenance tasks when the process is ending.

  • 📦 Zoom on thumbnail hover was disabled (to accommodate motion-sensitive users). If you miss this feature, please post a feature request to the forum to add this as a UI preference (much like thumbnail size).

  • 📦 logcat accepts file and directory paths now. Directories are recursively searched for “.log” and “.log.gz” files.

  • 📦 The command run by “open file in folder” is now customizable to support XFCE and other window managers. See openFileInFolderUsesFileUri and openFileInFolderCommand.

Prior release notes

Current release notes