Sign in

    photo_merge

    Review-first photo backup merge tool with exact and visual comparison

    photo
    backup
    image
    deduplication
    Download zip
    Author
    Version
    0.1.1
    License
    Apache-2.0
    Last updated
    12 hours ago
    Downloads
    3

    #照片备份合并预检

    一个用 MoonBit 编写的本地工具:先比较现有相册与待导入备份,生成可审核的 HTML 清单,再按确认的清单复制照片。适合合并多次备份产生的重复、近似和同名照片。

    项目只读取 JPEG、PNG,复制时不删除原件,也不覆盖现有文件。扫描与审核都在本机进行。

    #使用

    需要 MoonBit 工具链、Git 和 Node.js 24。已在 Windows 的 MoonBit 0.1.20261023、Node.js 24 及 GitHub Actions 的 Linux 环境运行。先获取仓库并进入项目目录:

    git clone https://github.com/ppyj663/moon-photo-merge.git cd moon-photo-merge

    核心 MoonBit 模块已发布至 Mooncakes,可在其他 MoonBit 项目中用 moon add ppyj663/photo_merge 引入。下面的命令行工具通过源码仓库运行。

    moon update moon run cmd/main -- scan "D:\Photos" "E:\PhoneBackup" "D:\PhotoReview"

    在浏览器打开输出的 review.html。页面会列出每张待导入图片及参考图片,并汇总各类数量;可以按文件名或路径搜索,也可按分类筛选。“只选可新增”可一次勾选明确的新照片,“清空选择”会取消所有勾选。“疑似相似”和“同名冲突”需手动勾选。下载 approved.json 后执行:

    moon run cmd/main -- check "D:\Photos" "E:\PhoneBackup" "C:\Users\你的用户名\Downloads\approved.json" moon run cmd/main -- apply "D:\Photos" "E:\PhoneBackup" "C:\Users\你的用户名\Downloads\approved.json"

    check 会重新扫描并验证审核清单,列出将复制的文件名与容量;它是只读预检,不改动照片。确认预检输出后再运行 apply。

    每次扫描还会在报告文件夹输出 review.csv、review.json、inventory.json、extensions.csv、similar-groups.json 和 similar-groups.csv。这些文件用于查看或二次分析;只有网页下载的 approved.json 是 apply 接受的审核清单。inventory.json 汇总容量、方向、像素、文件扩展名和大小区间;CSV 字段会转义分隔符、引号和换行,并给公式样式的文本字段添加单引号以便在电子表格中按文本打开。需要原始字段值时使用 JSON 导出。

    #常见使用场景

    1. 合并一次手机备份:按默认阈值扫描,打开 HTML 报告,保留“可新增”的默认选择,下载审核清单后执行 apply。
    2. 处理同名不同内容:报告会把文件标为“同名冲突”,给出带 -import-1 的建议文件名;查看两张预览后再决定是否勾选。
    3. 更敏感地查找近似照片:提高指纹差异阈值,例如 --similarity-limit 8,报告会把更多近似候选交给人工复核;扫描与执行必须使用相同阈值。

    阈值是 176 位视觉指纹的最大差异位数,范围为 0..176,默认值为 5。距离越小,指纹越接近;该分值不是相似概率。调低阈值会减少疑似提示,调高会增加提示。阈值不会代替人工审核。

    报告会显示图片尺寸、估算像素数、平均亮度和平均颜色的简单观察提示。工具还会按阈值把现有相册和备份中有直接匹配边的照片整理成“相似候选组”;组按直接匹配关系的连通性构成,因此同一组里的任意两张照片不一定彼此直接相似。候选组只用于缩小人工核对范围,不会自动排除或删除照片。

    moon run cmd/main -- scan "D:\Photos" "E:\PhoneBackup" "D:\PhotoReview" --similarity-limit 8 moon run cmd/main -- check "D:\Photos" "E:\PhoneBackup" "C:\Users\你的用户名\Downloads\approved.json" --similarity-limit 8 moon run cmd/main -- apply "D:\Photos" "E:\PhoneBackup" "C:\Users\你的用户名\Downloads\approved.json" --similarity-limit 8

    scan 要求三个文件夹彼此独立。check 和 apply 会重新扫描;若相册或备份内的 JPEG/PNG 内容、文件名、数量改变,旧清单就会被拒绝,需重新审核。check 只显示将复制的项目和容量。apply 只复制清单中勾选的文件到现有相册根目录,同名时使用 -import-1 等新文件名。复制后校验 SHA-256,且目标名已存在时停止,不会覆盖。

    命令帮助:moon run cmd/main -- --help。

    #判断规则

    分类含义默认选择
    完全重复文件字节 SHA-256 相同不可选择
    疑似相似可解码,亮度和颜色的简易视觉指纹接近不选择,人工核对
    同名冲突文件名相同但内容不同不选择,人工核对
    可新增未命中上述情况选择
    图片无法解码扩展名符合,但 JPEG/PNG 解码失败不可选择

    视觉指纹是提示,不保证识别所有重编码、裁剪或旋转后的照片,也可能误报。JPEG 的 EXIF 朝向不参与归一化。PNG 的解压数据上限为 128 MiB,超出上限、校验失败或无法解码的文件标为无法解码。亮度、颜色和分辨率提示是对采样图像的粗略统计,不等于图像质量判断。现有相册和备份的目录结构不会在目标中保留;目标文件名会被展开到相册根目录。首版不处理 HEIC、RAW、视频或云端照片。

    #示例与测试

    examples/fixtures 是合成测试素材,不含真实相册照片;其中包含五种待导入文件:完全重复、像素略变、同名不同内容、可新增 JPEG,以及损坏 JPEG。演示前复制它们到临时工作目录,避免修改仓库样例。跨平台运行方式:

    node scripts/demo.mjs

    脚本会打印临时报告路径。打开报告,下载清单,然后按上面的 apply 命令尝试复制。Windows 用户也可运行 ./scripts/demo.ps1。运行自动化测试:

    moon test --target js node scripts/integration-test.mjs

    核心分类、命名冲突、分段索引的候选召回、连通分组、统计边界、筛选分页、CSV/JSON 转义、损坏图片及重复导入的规则见测试;文件复制和报告导出流程可用演示数据复核。

    集成测试还覆盖空参数、空目录、阈值边界、嵌套目录、大写扩展名、零字节图片、PNG 分块及校验错误、较大 PNG、未支持格式、目录符号链接、路径穿越、过期清单、重复/不可复制清单项、目标文件碰撞、报告文件与原照片的硬链接,以及拒绝把基准汇总写进照片输入目录。扫描不会跟随符号链接目录;报告里可见的每个待导入路径都会在 check 和 apply 时重新校验。

    大赛验收项、仓库证据与本机复现命令见验收自查。

    #本地照片目录基准

    可用自己的照片目录测量完整 scan 命令。脚本先暖机一次,再按指定次数测量;每次报告写入系统临时目录并在结束时清理。汇总仅包含环境、文件数、字节数、分类数量和耗时,不写入路径、文件名、哈希或照片内容。扫描本身是只读的。

    node scripts/benchmark.mjs "D:\Photos" "E:\PhoneBackup" --runs 5 node scripts/benchmark.mjs "D:\Photos" "E:\PhoneBackup" --runs 5 --output "D:\PhotoReview\benchmark-summary.json"

    基准报告中的 medianMs 是重复运行中位数,p95Ms 使用最近秩法;耗时包括 CLI 启动、扫描、解码、分类、分组和报告导出。建议固定设备、目录和 MoonBit/Node.js 版本后比较。examples/fixtures 是便于 CI 复现的合成夹具,性能结果不能当作真实相册的性能或识别准确率承诺;上面的命令可直接生成真实照片目录的本机汇总。

    当前 Windows 10 x64 参考运行(Node.js 24.19.0、MoonBit 0.1.20261023,2026-10-28)使用仓库合成夹具,现有相册 1 张/198 B,待导入 5 张/1,212 B;5 次测量中位数 399.306 ms,p95 为 477.864 ms。五张待导入样例各覆盖完全重复、疑似相似、同名冲突、可新增和无法解码一种分类。完整汇总见 基准 JSON。该小样本主要验证基准流程与报告统计,进程启动开销占比高,不用于推断大相册性能。

    另用 scikit-image 的 6 张公开照片构造两组输入,测量真实 JPEG/PNG 解码与扫描:7 个文件时 5 次中位数为 787.364 ms;把同一组照片复制扩展到 70 个文件时中位数为 3,855.594 ms。扩展组只有 6 张不同原图,不能视为 70 张不同照片或真实用户相册。数据来源、生成方法、全部测量和局限见公开照片基准。

    #实现

    MoonBit 负责照片数据建模、图片解码后的视觉指纹、分类和导入决策。JPEG/PNG 解码使用 gmlewis/image,文件系统及 SHA-256 使用 Node.js 适配代码。PNG 使用 Node.js zlib 校验并转换压缩数据后交给 MoonBit 解码,以避免依赖库在 JS 目标上发生深层递归;像素不因此改变。报告通过临时文件替换,避免跟随已有输出文件的硬链接或符号链接改写其目标。执行阶段再次核对扫描摘要和来源文件哈希,通过同目录临时文件校验后创建目标硬链接,因此不会覆盖同名文件;若执行中途失败,已成功复制的文件会保留,重新扫描后可继续审核。

    直接依赖为 gmlewis/image 和 gmlewis/io;传递依赖与上游来源说明见第三方来源。本项目使用了 Codex 辅助编写代码、测试和文档;分类规则、限制与复现方式见上文及仓库测试。

    许可证:Apache-2.0。

    DigestGroup

    pub(all) struct DigestGroup {
    digest : String
    paths : Array[String]
    bytes : Int64
    }

    ExtensionStats

    pub(all) struct ExtensionStats {
    extension : String
    count : Int
    readable : Int
    unreadable : Int
    bytes : Int64
    }

    ImageInfo

    pub(all) struct ImageInfo {
    width : Int
    height : Int
    luminance : Int
    red : Int
    green : Int
    blue : Int
    }

    Basic decoded-image facts used for inventory and human review.

    InventorySummary

    pub(all) struct InventorySummary {
    total : Int
    readable : Int
    unreadable : Int
    bytes : Int64
    minimum_size : Int
    maximum_size : Int
    average_size : Int64
    landscape : Int
    portrait : Int
    square : Int
    unknown_dimensions : Int
    pixels : Int64
    duplicate_file_groups : Int
    duplicate_files : Int
    }

    Kind

    pub(all) enum Kind {
    Exact
    Similar
    Conflict
    New
    Unreadable
    } derive(Eq)

    Kind::code

    fn Kind::code(self : Kind) -> String

    Kind::equal

    fn Kind::equal(Kind, Kind) -> Bool

    Kind::label

    fn Kind::label(self : Kind) -> String

    Kind::not_equal

    fn Kind::not_equal(x : Kind, y : Kind) -> Bool

    Photo

    pub(all) struct Photo {
    path : String
    name : String
    digest : String
    fingerprint : String?
    size : Int
    image_info : ImageInfo?
    }

    Facts about one regular JPEG/PNG file. Missing fingerprint means decoding failed.

    PlanSummary

    pub(all) struct PlanSummary {
    total : Int
    exact : Int
    similar : Int
    conflict : Int
    new_count : Int
    unreadable : Int
    copyable : Int
    needs_review : Int
    selected_by_default : Int
    total_bytes : Int64
    copyable_bytes : Int64
    review_bytes : Int64
    }

    ReviewPage

    pub(all) struct ReviewPage {
    rows : Array[ReviewRow]
    total_matches : Int
    offset : Int
    page_size : Int
    page_count : Int
    has_previous : Bool
    has_next : Bool
    }

    ReviewQuery

    pub(all) struct ReviewQuery {
    text : String
    kind : Kind?
    minimum_size : Int?
    maximum_size : Int?
    sort : ReviewSort
    offset : Int
    page_size : Int
    }

    ReviewRow

    pub(all) struct ReviewRow {
    path : String
    name : String
    digest : String
    kind : Kind
    match_path : String?
    similarity_distance : Int?
    destination : String?
    size : Int
    image_info : ImageInfo?
    }

    ReviewSort

    pub(all) enum ReviewSort {
    PathAscending
    NameAscending
    SizeDescending
    SimilarityAscending
    CategoryThenPath
    } derive(Eq)

    Sort order available to the local review page and other consumers.

    ReviewSort::equal

    fn ReviewSort::equal(ReviewSort, ReviewSort) -> Bool

    ReviewSort::not_equal

    fn ReviewSort::not_equal(x : ReviewSort, y : ReviewSort) -> Bool

    SimilarityGroup

    pub(all) struct SimilarityGroup {
    paths : Array[String]
    pair_count : Int
    minimum_distance : Int
    maximum_distance : Int
    }

    A connected review set of photos joined by at least one qualifying visual fingerprint edge. Members at the ends of a chain need not match directly.

    SimilarityGroupsSummary

    pub(all) struct SimilarityGroupsSummary {
    groups : Int
    members : Int
    pairs : Int
    largest_group : Int
    }

    SimilarityHit

    pub(all) struct SimilarityHit {
    path : String
    name : String
    digest : String
    distance : Int
    }

    One possible visual match found by the Hamming-distance index.

    SimilarityIndex

    pub struct SimilarityIndex {
    max_distance : Int
    entries : Array[Photo]
    buckets64 : Array[Map[String, Array[Int]]]
    buckets176 : Array[Map[String, Array[Int]]]
    }

    Segmented signature index. A signature within the configured Hamming limit must share at least one complete segment, so each lookup examines only buckets that can contain a qualifying candidate and verifies the full distance before returning it.

    SimilarityIndex::add

    fn SimilarityIndex::add(self : SimilarityIndex, photo : Photo) -> Unit

    Add a decodable photo to the index. Unsupported or absent signatures are ignored and cannot be returned as visual candidates.

    SimilarityIndex::all_pairs

    fn SimilarityIndex::all_pairs(self : SimilarityIndex, limit : Int) -> Array[(String, String, Int)]

    Count every qualifying unordered photo pair in an already-created index. Each pair is returned once, in stable path order.

    SimilarityIndex::capacity

    fn SimilarityIndex::capacity(self : SimilarityIndex) -> Int

    Largest distance for which this index guarantees candidate recall.

    SimilarityIndex::count

    fn SimilarityIndex::count(self : SimilarityIndex) -> Int

    Number of decodable photos currently indexed.

    SimilarityIndex::nearest

    fn SimilarityIndex::nearest(self : SimilarityIndex, signature : String, limit : Int) -> SimilarityHit?

    Return the closest qualifying match. Equal distances resolve by path so shuffled input order cannot change the selected reference.

    SimilarityIndex::search

    fn SimilarityIndex::search(self : SimilarityIndex, signature : String, limit : Int) -> Array[SimilarityHit]

    Return all candidates at or below a requested distance, ordered by distance and then path. Results do not claim that files depict the same scene.

    SizeBand

    pub(all) enum SizeBand {
    Under256KiB
    From256KiBTo1MiB
    From1MiBTo8MiB
    From8MiBTo32MiB
    AtLeast32MiB
    } derive(Eq)

    A byte-size band used by the inventory summary.

    SizeBand::equal

    fn SizeBand::equal(SizeBand, SizeBand) -> Bool

    SizeBand::label

    fn SizeBand::label(self : SizeBand) -> String

    SizeBand::not_equal

    fn SizeBand::not_equal(x : SizeBand, y : SizeBand) -> Bool

    SizeBandStats

    pub(all) struct SizeBandStats {
    band : SizeBand
    count : Int
    bytes : Int64
    }

    analyze_image

    fn analyze_image(data : Bytes, extension : String) -> (String, ImageInfo)?

    Difference hash, average luminance and coarse mean colors. A close hash is only a review hint and never authorizes deletion.

    build_plan

    fn build_plan(existing : Array[Photo], incoming : Array[Photo], similar_limit? : Int) -> Array[ReviewRow]

    Similarity is a review hint; only identical bytes are auto-skipped.

    csv_field

    fn csv_field(value : String) -> String

    Quote every CSV field so commas, quotes, line breaks and spreadsheet text are preserved as data. Formula-like text receives a leading apostrophe for safer opening in spreadsheet software; JSON remains the exact-value export.

    default_review_query

    fn default_review_query() -> ReviewQuery

    Default query for a fresh report view: all rows by original path, first 100 entries. Pagination is available to consumers that render large scans.

    duplicate_digest_groups

    fn duplicate_digest_groups(photos : Array[Photo]) -> Array[DigestGroup]

    Return groups of byte-identical files. Group order follows first appearance in the input, and path order follows the input order as well.

    extension_summary

    fn extension_summary(photos : Array[Photo]) -> Array[ExtensionStats]

    Aggregate photo count, decoded count and bytes by normalized extension. Output is sorted alphabetically for reproducible reports.

    extension_summary_csv

    fn extension_summary_csv(items : Array[ExtensionStats]) -> String

    Export aggregate extension totals as a spreadsheet-friendly CSV.

    find_similarity_group

    fn find_similarity_group(groups : Array[SimilarityGroup], path : String) -> SimilarityGroup?

    Find the connected group containing a path, if it has a qualifying edge.

    fingerprint

    fn fingerprint(data : Bytes, extension : String) -> String?

    Build the 176-bit image signature when the input is a supported decodable image.

    format_bytes

    fn format_bytes(bytes : Int64) -> String

    Format a byte count using binary units. The output is intended for reports, not as a machine-readable value.

    hamming_distance

    fn hamming_distance(a : String, b : String) -> Int

    image_info_label

    fn image_info_label(info : ImageInfo) -> String

    Present decoded dimensions and approximate megapixels in one short label.

    image_review_notes

    fn image_review_notes(info : ImageInfo) -> Array[String]

    Explain a few visible characteristics that can help a person compare a questionable file. These are observations, not quality verdicts.

    inventory_json

    fn inventory_json(summary : InventorySummary, extensions : Array[ExtensionStats], size_bands : Array[SizeBandStats]) -> String

    Export a concise scan inventory as JSON for scripts and audit tools.

    inventory_summary

    fn inventory_summary(photos : Array[Photo]) -> InventorySummary

    Summarize a folder scan without changing the photos or their order.

    json_string

    fn json_string(value : String) -> String

    Quote a string as a JSON value, including control characters and Unicode.

    new_similarity_index

    fn new_similarity_index(max_distance : Int) -> SimilarityIndex

    Create an index whose guarantee applies up to max_distance differing bits. The supported signature widths are 64 and 176 bits.

    normalized_extension

    fn normalized_extension(name : String) -> String

    Normalize path separators before extracting a final component. The collection API otherwise treats paths as opaque labels.

    plan_summary

    fn plan_summary(rows : Array[ReviewRow]) -> PlanSummary

    Counts and byte totals for the incoming review plan.

    query_review_rows

    fn query_review_rows(rows : Array[ReviewRow], query : ReviewQuery) -> ReviewPage

    Filter, sort and page plan rows. Negative size bounds are treated as zero; an inverted range returns an empty page. Page sizes are clamped to 1–500.

    review_page_number

    fn review_page_number(offset : Int, page_size : Int) -> Int

    Report the one-based page containing a row offset.

    review_page_start

    fn review_page_start(page_number : Int, page_size : Int) -> Int

    A lightweight helper for consumers that want to show page numbers.

    review_rows_csv

    fn review_rows_csv(rows : Array[ReviewRow]) -> String

    Export decisions as RFC-style quoted CSV for spreadsheet review.

    review_rows_json

    fn review_rows_json(rows : Array[ReviewRow]) -> String

    Export the full decision table as a UTF-8 JSON array. This is a report artifact; it is not an approval file and cannot be passed to apply.

    similarity_groups

    fn similarity_groups(photos : Array[Photo], limit : Int) -> Array[SimilarityGroup]

    Convert qualifying pairs into connected review groups. Group membership is transitive by design and is not a claim that every pair in a group is within the configured distance.

    similarity_groups_csv

    fn similarity_groups_csv(groups : Array[SimilarityGroup]) -> String

    Export each candidate photo together with its connected group summary.

    similarity_groups_json

    fn similarity_groups_json(groups : Array[SimilarityGroup]) -> String

    Export path-sorted connected groups. Consumers should display the rule note because transitive membership does not mean every pair is close.

    similarity_groups_summary

    fn similarity_groups_summary(groups : Array[SimilarityGroup]) -> SimilarityGroupsSummary

    Summarize connected candidate groups for a report header.

    similarity_pairs_csv

    fn similarity_pairs_csv(pairs : Array[(String, String, Int)]) -> String

    Export direct qualifying edges as CSV for use in a spreadsheet.

    size_band

    fn size_band(size : Int) -> SizeBand

    size_distribution

    fn size_distribution(photos : Array[Photo]) -> Array[SizeBandStats]

    Count files and total bytes in fixed, documented size ranges.