Author Topic: The Deduplication Project  (Read 2890 times)

0 Members and 1 Guest are viewing this topic.

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
The Deduplication Project
« on: June 03, 2017 - 20:19:44 »
~

« Last Edit: August 27, 2017 - 20:37:09 by attractivo »

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #1 on: June 07, 2017 - 10:12:17 »
~
« Last Edit: August 27, 2017 - 20:37:26 by attractivo »

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #2 on: June 08, 2017 - 20:13:37 »
~
« Last Edit: August 27, 2017 - 20:37:47 by attractivo »

Offline Nukhem

  • Newbie
  • *
  • Posts: 47
Re: The Deduplication Project
« Reply #3 on: June 09, 2017 - 22:37:58 »
Amazing work ! This cleaned up my 1,4TB ToSort unknown files folder alot :)

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #4 on: June 10, 2017 - 01:15:59 »
glad to hear that  :)

« Last Edit: August 27, 2017 - 20:38:11 by attractivo »

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #5 on: June 15, 2017 - 23:59:38 »
~
« Last Edit: August 27, 2017 - 20:38:19 by attractivo »

Offline dizzzy

  • Full Member
  • ***
  • Posts: 123
Re: The Deduplication Project
« Reply #6 on: June 16, 2017 - 01:09:19 »
20161226        *\Sony - PlayStation *\* [bsbt]

What's this? I've been working on a companion set to PSX (basically 900 unique games not yet in redump.org). Would love more info on the source of this torrent(?)

Edit: just noticed Rom Shepherd has a tracker, I guess it's on there. Needa get on.
« Last Edit: June 16, 2017 - 01:12:44 by user7 »
Non-Redump PSX set (all games not dumped to redump.org) [You are not allowed to view links] Register or Login
Message me if you have anything to add.

Recent PSX Redumps: [You are not allowed to view links] Register or Login

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #7 on: June 16, 2017 - 01:52:59 »
those are PSN and Firmware collections for playstation systems datted by 'bsbt' which can be found on the nointro forum site.

Offline Connie

  • Hero Member
  • *****
  • Posts: 1866
Re: The Deduplication Project
« Reply #8 on: June 16, 2017 - 01:54:30 »
[You are not allowed to view links] Register or Login
What's this? I've been working on a companion set to PSX (basically 900 unique games not yet in redump.org). Would love more info on the source of this torrent(?)
Redump doesn't = Scene
Your 900 games is probably a mix of bad and not-redumped.
...or something that isn't disc based and is therefore in another dat.

EDIT:
Ninja'd by @tractivo
"Get busy living or get busy dying" - Shawshank Redemption (Stephen King)

My DAT Files - [You are not allowed to view links] Register or Login
My Shared Files - [You are not allowed to view links] Register or Login
My GOG.com Files - [You are not allowed to view links] Register or Login

Offline dizzzy

  • Full Member
  • ***
  • Posts: 123
Re: The Deduplication Project
« Reply #9 on: June 16, 2017 - 04:36:28 »
[You are not allowed to view links] Register or Login
those are PSN and Firmware collections for playstation systems datted by 'bsbt' which can be found on the nointro forum site.

Gotcha, thanks for clearing that up.


[You are not allowed to view links] Register or Login
Redump doesn't = Scene
Your 900 games is probably a mix of bad and not-redumped.
...or something that isn't disc based and is therefore in another dat.

I know redump isn't scene, I dump there. My rom collection is PSX discs floating around on the internet that have not yet been redumped. Mostly rare japanese titles from Russian sites. But I was out of the loop about the bsbt no-intro set and thought that might be something similar to what I compiled (I understand now that it's not).
« Last Edit: June 16, 2017 - 04:39:51 by user7 »
Non-Redump PSX set (all games not dumped to redump.org) [You are not allowed to view links] Register or Login
Message me if you have anything to add.

Recent PSX Redumps: [You are not allowed to view links] Register or Login

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #10 on: June 17, 2017 - 20:42:01 »
~
« Last Edit: August 27, 2017 - 20:38:40 by attractivo »

Offline rzil

  • Newbie
  • *
  • Posts: 21
Re: The Deduplication Project
« Reply #11 on: June 17, 2017 - 20:44:25 »
Wow, great project!! Thanks.

Can I ask what does "_Old" folder means? (duplicates, deprecated, something else?)

Also, I suggest moving GoodSets and TOSEC to be the very last in source order...
both are extremely large and shadowing the names of other collections,
which often have much more meaningful names.
« Last Edit: June 17, 2017 - 20:47:42 by rzil »

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: The Deduplication Project
« Reply #12 on: June 18, 2017 - 01:11:15 »
~
« Last Edit: August 27, 2017 - 20:38:54 by attractivo »

Offline rzil

  • Newbie
  • *
  • Posts: 21
Re: The Deduplication Project
« Reply #13 on: June 18, 2017 - 05:43:16 »
Thanks for the explanation.

For instance, Super Mario 64 Hacks which are listed by name and creator in Yori's NonGoods
are all named "Super Mario 64 (1996)(Nintendo)(US)[hXXX]" in TOSEC

I would sort by specifity/speciality (accuracy is also very specific [only most accurate ROMs]), means DATs with very special goal will be high (first) in order (not much will change).
for example Maybe-Intro, which is specific to [SNES] rom translations will be before GoodSets, which tries to list every possible version.
and Zandro's SMW Hacks, which is specific to SMB Hacks, even before Maybe-Intro.
another criteria is meaningful naming (altough I think naming and speciallity often come together 8))

of course it is just a suggestion and you can do what you think is best.

Offline cannonwillow

  • Jr. Member
  • **
  • Posts: 85
Re: The Deduplication Project
« Reply #14 on: June 18, 2017 - 07:28:07 »
thanks for all the great work @Tractivo. Do you plan on Deduping the Dats2\Artwork dats? there's a lot of duplicate roms, especially in the \progetto-SNAPS folder.

cannonwillow