folder deduper

 If like me you have thousands of duplicate photos, you need this. it only deduplicates on md5 (identical) files, not visually similar files. It does not run inside github as that will break any coding you are doing.


#!/bin/bash
# dupemd5.sh — report duplicate files by MD5; dry-run move + CSV log

# Safety check: do not allow running inside GitHub directories
if [[ "$(pwd)" =~ [Gg]ithub ]]; then
    echo "Error: refusing to run inside a GitHub directory ($(pwd))" >&2
    exit 1
fi

APPLY=0
if [[ "${1:-}" == "--apply" ]]; then
    APPLY=1
    shift
fi
dir="${1:-.}"

declare -A seen
count=0

duplicates_dir="./duplicates"
csv="$duplicates_dir/duplicates_found.csv"

mkdir -p "$duplicates_dir"
touch "$csv"
if [[ ! -s "$csv" ]]; then
    echo '"item";"original";"duplicate";"target"' > "$csv"
fi

print_summary() {
    echo "Total duplicate pairs found: $count"
}

log_csv() { # idx, orig, dup, target
    local idx="$1" orig="$2" dup="$3" target="$4"
    esc() { printf '%s' "$1" | sed 's/"/""/g'; }
    printf '"%s";"%s";"%s";"%s"\n' \
        "$(esc "$idx")" "$(esc "$orig")" "$(esc "$dup")" "$(esc "$target")" >> "$csv"
}

unique_name() {
    local target="$1"
    local base="$(basename "$target")"
    local name="${base%.*}"
    local ext="${base##*.}"
    local dir="$(dirname "$target")"
    local n=1
    local candidate="$target"
    while [[ -e "$candidate" ]]; do
        n=$((n+1))
        if [[ "$ext" != "$name" ]]; then
            candidate="$dir/${name}_$n.$ext"
        else
            candidate="$dir/${name}_$n"
        fi
    done
    echo "$candidate"
}

move_dup() { # dry-run by default; real move with --apply
    local dup="$1" target="$2"
    target=$(unique_name "$target")
    if (( APPLY )); then
        mv -n -- "$dup" "$target"
    else
        echo "mv -n -- \"$dup\" \"$target\""
    fi
    echo "$target"
}

while IFS= read -r -d '' file; do
    md5=$(md5sum "$file" | awk '{print $1}')
    if [[ -n "${seen[$md5]}" ]]; then
        echo "original: ${seen[$md5]}"
        echo "apparent duplicate: $file"
        echo
        echo

        orig="${seen[$md5]}"
        dup="$file"
        target="$duplicates_dir/$(basename "$dup")" 

        orig_abs="$(realpath -m "$orig")"
        dup_abs="$(realpath -m "$dup")"

        target_abs="$(move_dup "$dup_abs" "$target")"
        count=$((count+1))
        log_csv "$count" "$orig_abs" "$dup_abs" "$target_abs"
    else
        seen[$md5]="$file"
    fi
done < <(find "$dir" -type f -print0 | LC_ALL=C sort -z)

print_summary

The following is the undo script: make sure you are in the directory that was deduped, do not go into the duplicates directory first, as it tries to read duplicates/duplicates_found.csv to undo the moves.

#!/bin/bash
# undo_dupes.sh — restore moved duplicates back to their original locations
# Usage:
#   bash undo_dupes.sh              # dry-run
#   bash undo_dupes.sh --apply      # actually move files

APPLY=0
if [[ "${1:-}" == "--apply" ]]; then
    APPLY=1
    shift
fi

csv="${1:-./duplicates/duplicates_found.csv}"

if [[ ! -f "$csv" ]]; then
    echo "CSV not found: $csv" >&2
    exit 1
fi

awk -v FPAT='([^;]*)|(\"([^\"]|\"\")*\")' '
NR==1 { next }  # skip header
function unq(s){ if (s ~ /^".*"$/){ s=substr(s,2,length(s)-2); gsub(/""/,"\"",s) } return s }
{
  dup = unq($3)   # duplicate original path (destination to restore to)
  tgt = unq($4)   # where the file was moved (source to restore from)
  print dup "\t" tgt
}
' "$csv" | while IFS=$'\t' read -r dup tgt; do
    if [[ -f "$tgt" ]]; then
        if (( APPLY )); then
            mkdir -p -- "$(dirname "$dup")"
            mv -n -- "$tgt" "$dup"
            echo "restored: $dup"
        else
            echo "mv -n -- \"$tgt\" \"$dup\""
        fi
    else
        echo "missing (skipped): $tgt" >&2
    fi
done

Popular posts from this blog

Automatically Fix Song Metadata and Filenames on Linux with Beets

 🎵 Automatically Fix Song Metadata and Filenames on Linux with Beets Tired of sorting through unknown MP3s with cryptic filenames like Track01.mp3? Sick of missing album info, incorrect genres, and zero cover art? Here's a clean way to batch-fix your music collection with open source tools — no manual editing required. ✅ What You’ll Use beets — the core tool that organizes, renames, and tags your music by matching against the MusicBrainz database. libchromaprint-tools — provides fpcalc, which generates audio fingerprints to identify unlabeled tracks. ffmpeg (optional) — used by beets for transcoding or embedded cover art processing. 🔧 Installation sudo apt update sudo apt install beets libchromaprint-tools ffmpeg 🚀 One Command to Organize It All beet import /path/to/your/music/ Beets will: Fingerprint each file Query MusicBrainz for metadata Rename files (e.g., Artist - Title.mp3) Write correct ID3/metadata tag Optionally move files into organized folders 📁 Example Output Your ...
Read more

throttle traffic on apache

sudo apt install apache2-utils sudo apt install libapache2-mod-evasive vi /etc/apache2/mods-enabled/evasive.conf service apache restart edit it and set values as per below or whatever else you like, the time quantities are seconds     DOSSiteInterval     1     DOSBlockingPeriod   10     #DOSLogDir           "/var/lock/mod_evasive"
Read more

script to show all files with disk usage sizes

Normally to get the disk usage of a folder you use get properties on the rightclick or similar. However there are hidden files starting with . in their name that you can get info on, only in the terminal, using the  du -sh  command. If you have hundreds of them (which you do!) this becomes tedious. The following script gives you a directory listing with the files all indicated in file sizes and highlights on files of megabytes or gigabytes in size.  #!/bin/bash # Set Internal Field Separator to handle spaces in filenames IFS=$'\n' # Loop through all files and directories for i in $(/bin/ls -A); do        size=$(du -sh "$i" | awk '{print $1}')           # Apply colors: bold for MB, bright bold for GB      case "$size" in          *M) color="\e[1m" ;;         # Bold for Megabytes          *G) color="\e[1;97m" ;;      # Br...
Read more