What do you mean by "deduplicate"?

What do you mean by "deduplicate"?

Browsing with PhotoStructure is designed to be fast and fun.

As you navigate through your photos and videos, and you have duplicate photos or videos, clicking “next” or “previous” can result in seeing the same thing. But wait: did you not click the button? Is this a bug? Either way: these browsing stutters aren’t fun.

To avoid this, PhotoStructure automatically detects duplicate photo and video variations, and only shows you the “best” variant.

👥 Why you may have duplicates

There are several reasons why you might have 2 or more copies or variations of any given photo or video:

📷 RAW+JPEG pairs

Most current digital cameras and even some smartphones support “shooting raw.”

These raw files encode higher sensor fidelity than JPEGs. This additional information can allow you to “post-process” files to get better dynamic range, restore highlight and shadow details, and adjust color balance, with much more flexibility than an JPEG.

Unfortunately, raw images are slow to process, and many image applications can’t handle these files. Most cameras allow shooting in “RAW+JPEG,” where each time you push the shutter button, a JPEG file as well as a RAW image file is written to your memory card. If PhotoStructure didn’t know that these are actually the same image, you’d see two (or more) photos with the same image while browsing your PhotoStructure library.

☁️ Cloud backups

Several photo cloud backup services downsample your photos and videos, and strip much of the metadata from your files, as well.

If you download a local backup from your cloud service, these photos and videos will be duplicates of your original files.

✏️ Local edits

When you make edits to your images, some software will write to a new file rather than overwriting your original.

💾 Local backups

If you’ve used backup software you’ll have several copies of your photos and videos where the backup destination was configured.

🗂️ How this relates to automatic organization

If you’ve enabled automatic organization, PhotoStructure errs on the side of caution, and copies each valid, unique image into your library.

If exactly the same file is found (i.e., precisely the same stream of bytes on disk), it won’t be copied into your library again. All other variants to the image, though, will be copied.

As an example, in the above cases, both the raw and JPEG files will be copied into your library, as well as any unique files from cloud service backups, and local edits.

🔗 How files are aggregated

A number of metadata tags are examined in each file, and if both files have a value for a given tag, and they substantively differ, the files are considered to be different assets.

If the captured-at time matches, but an insufficient number of other metadata tags match, PhotoStructure will compare the actual images of the files. If they are substantively different, the files are considered to be different assets.

You can use the info tool to compare files and see if PhotoStructure considers them eligible to be associated to the same asset.

🔍 Similarity search for fuzzy-dated images

When photos have precise timestamps, PhotoStructure can confidently match duplicates by comparing captured-at times and other metadata. But many photos (especially scanned images or files from cloud backups that stripped metadata) only have “fuzzy” dates inferred from directory names like 2019-Summer-Vacation/.

For these files, PhotoStructure uses perceptual image hashing to detect duplicates. Unlike a cryptographic hash (which changes completely if a single pixel differs), perceptual hashes capture the visual essence of an image. Two photos that look the same will have similar hashes, even if they differ in resolution, compression, or metadata.

PhotoStructure stores these hashes in a sqlite-vec vector index, enabling fast similarity searches across your entire library. When a fuzzy-dated file is imported, PhotoStructure searches for visually similar images within the date range and groups them as variants of the same asset.

See allowFuzzyDateImageHashMatches and related imageHash* settings in library settings to tune this behavior.

🏆 How does PhotoStructure pick which file to show?

In general, PhotoStructure picks the “best” image or video variation with the largest resolution that lives in your library.

In an effort to make PhotoStructure’s “best” pick be predictable, though, there are a number of other file metadata attributes that PhotoStructure also uses. The variantSortCriteria library setting allows you to customize how PhotoStructure picks your library’s “best”.

Here’s the list of those fields, in default priority order, as of v2024:

  • resolution: the coarse image resolution. Similar megapixel resolution differences are considered equivalent.

  • schemeIdx: captures “where the file resides” (it references the asset file URI scheme). This prefers files stored in your library over files found outside your library, and prefers files stored on a volume with a volume UUID over files on volumes that do not have a UUID.

  • capturedAtPrecision: variations that contain more reliable captured-at metadata will be preferred.

  • metadataDate: When PhotoStructure (and some other applications) make edits to metadata, the MetadataDate tag will be updated. Files with a more recent MetadataDate will be preferred.

  • metadataCoverage: prefer files with more fields with metadata we care about

  • isBrowserSupported: prefer files we can directly stream to the browser without re-rendering or transcoding

  • isEditOrUpdate: prefer files whose basename includes “edit” or “update”. Many editing applications will save “file-updated.jpg” instead of overwriting the original file.

  • isCover: if we have a burst files, prefer the “burst cover”

  • count: if there are many copies of a file (image.jpg, image (1).jpg, image (2).jpg), prefer the one with the highest number (assuming that’s the latest copy)

  • fileSize: prefer larger files. This may (or may not) be indicative of less compression loss, but also gives us one more attribute before we resort to the (noisy and fairly unreliable) mtime attribute.

  • mtime: prefer that newest version. Note that many backup applications don’t retain mtime correctly, so we don’t really trust this value

  • basename: this helps make sorting deterministic if all other factors are the same

  • parentBasename: this helps make sorting deterministic

  • uri: this is simply to make sorting deterministic if all other factors are the same