---
title: What do you mean by "deduplicate"?
url: https://photostructure.com/guide/what-do-you-mean-by-deduplicate/
description: What de-duplication means to PhotoStructure.
date: 2020-07-11
keywords: deduplication, RAW, JPEG, metadata, variants
---


**Browsing with PhotoStructure is designed to be fast and fun.**

As you navigate through your photos and videos, and you have duplicate photos or
videos, clicking "next" or "previous" can result in seeing the same thing. But
wait: did you not click the button? Is this a bug? Either way: these browsing
stutters aren't _fun_.

To avoid this, PhotoStructure automatically detects duplicate photo and video
variations, and only shows you the “best” variant.

## 👥 Why you may have duplicates

There are several reasons why you might have 2 or more copies or variations of
any given photo or video:

### 📷 RAW+JPEG pairs

Most current digital cameras and even some smartphones support "shooting raw."

These raw files encode higher sensor fidelity than JPEGs. This additional
information can allow you to “post-process” files to get better dynamic range,
restore highlight and shadow details, and adjust color balance, with much more
flexibility than an JPEG.

Unfortunately, raw images are slow to process, and many image applications
can't handle these files. Most cameras allow shooting in “RAW+JPEG,” where
each time you push the shutter button, a JPEG file as well as a RAW image file
is written to your memory card. If PhotoStructure didn’t know that these are
actually the same image, you’d see two (or more) photos with the same image
while browsing your PhotoStructure library.

### ☁️ Cloud backups

Several photo cloud backup services downsample your photos and videos, and strip
much of the metadata from your files, as well.

If you download a local backup from your cloud service, these photos and videos
will be duplicates of your original files.

### ✏️ Local edits

When you make edits to your images, some software will write to a new file
rather than overwriting your original.

### 💾 Local backups

If you've used backup software you'll have several copies of your photos and
videos where the backup destination was configured.

## 🗂️ How this relates to automatic organization

If you've enabled [automatic
organization](/getting-started/automatic-library-organization/), PhotoStructure
errs on the side of caution, and **copies** each valid, unique image into your
library.

If **exactly the same file** is found (i.e., precisely the same stream of bytes
on disk), it won’t be copied into your library again. All other variants to the
image, though, **will be copied**.

As an example, in the above cases, both the raw and JPEG files will be copied
into your library, as well as any unique files from cloud service backups, and
local edits.

## 🔗 How files are aggregated

A number of metadata tags are examined in each file, and if both files have a
value for a given tag, and they substantively differ, the files are considered
to be different assets.

If the captured-at time matches, but an insufficient number of other metadata
tags match, PhotoStructure will compare the actual images of the files. If they
are substantively different, the files are considered to be different assets.

You can [**use the `info` tool**](/server/tools/#file-comparisons) to compare files
and see if PhotoStructure considers them eligible to be associated to the same
asset.

<a id="similarity-search"></a>

## 🔍 Similarity search for fuzzy-dated images

When photos have precise timestamps, PhotoStructure can confidently match duplicates by comparing captured-at times and other metadata. But many photos (especially scanned images or files from cloud backups that stripped metadata) only have "fuzzy" dates [inferred from directory names](/faq/captured-at/#step-5-fuzzy-date-parsing) like `2019-Summer-Vacation/`.

For these files, PhotoStructure uses **perceptual image hashing** to detect duplicates. Unlike a cryptographic hash (which changes completely if a single pixel differs), perceptual hashes capture the visual essence of an image. Two photos that _look_ the same will have similar hashes, even if they differ in resolution, compression, or metadata.

PhotoStructure stores these hashes in a [sqlite-vec](https://github.com/mceachen/sqlite-vec) vector index, enabling fast similarity searches across your entire library. When a fuzzy-dated file is imported, PhotoStructure searches for visually similar images within the date range and groups them as variants of the same asset.

See `allowFuzzyDateImageHashMatches` and related `imageHash*` settings in [library settings](/getting-started/advanced-settings/#library-settings) to tune this behavior.

<a id="variantSortCriteria"></a>

## 🏆 How does PhotoStructure pick which file to show?

In general, PhotoStructure picks the "best" image or video variation with the
largest resolution that lives in your library.

In an effort to make PhotoStructure's "best" pick be predictable, though,
there are a number of other file metadata attributes that PhotoStructure also
uses. The `variantSortCriteria` [library
setting](/getting-started/advanced-settings/#library-settings) allows you to
customize how PhotoStructure picks your library's "best".

Here's the list of those fields, in default priority order, as of v2024:

- `resolution`: the coarse image resolution. Similar megapixel resolution
  differences are considered equivalent.

- `schemeIdx`: captures “where the file resides” (it references the asset file
  URI scheme). This prefers files stored in your library over files found
  outside your library, and prefers files stored on a volume with a [volume
  UUID](/faq/what-is-a-volume/#logical-volume-uuids)
  over files on volumes that do not have a UUID.

- `capturedAtPrecision`: variations that contain more reliable captured-at
  metadata will be preferred.
- `metadataDate`: When PhotoStructure (and some other applications) make edits
  to metadata, the `MetadataDate` tag will be updated. Files with a more recent
  `MetadataDate` will be preferred.

- `metadataCoverage`: prefer files with more fields with metadata we care
  about

- `isBrowserSupported`: prefer files we can directly stream to the browser
  without re-rendering or transcoding

- `isEditOrUpdate`: prefer files whose basename includes "edit" or "update".
  Many editing applications will save "file-updated.jpg" instead of
  overwriting the original file.

- `isCover`: if we have a burst files, prefer the "burst cover"

- `count`: if there are many copies of a file (image.jpg, image (1).jpg, image
  (2).jpg), prefer the one with the highest number (assuming that's the latest
  copy)
- `fileSize`: prefer larger files. This may (or may not) be indicative of less
  compression loss, but also gives us one more attribute before we resort to the
  (noisy and fairly unreliable) `mtime` attribute.

- `mtime`: prefer that newest version. Note that many backup applications
  don't retain `mtime` correctly, so we don't really trust this value
- `basename`: this helps make sorting deterministic if all other factors are
  the same

- `parentBasename`: this helps make sorting deterministic

- `uri`: this is simply to make sorting deterministic if all other factors are
  the same