Author Topic: Set Sprucing  (Read 4480 times)

0 Members and 1 Guest are viewing this topic.

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #15 on: June 16, 2017 - 07:09:10 »
Nope, it does not match the hash I specified.  It wasn't whether GoodGBA accepts it, it's HOW it's accepted.

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #16 on: June 16, 2017 - 08:35:44 »
Doesn't it use SHA-1 per default?
And why is it accepted when it doesn't match?

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #17 on: June 16, 2017 - 15:30:00 »
I heard Good insight and found CUE's bag of packers.  The ROM is LZSS encoded.  Decoding gives the correct hash.
« Last Edit: June 16, 2017 - 15:31:36 by Zandro »

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #18 on: June 16, 2017 - 16:08:38 »
I've just tested GoodSNES with both Dennis ROMs.
I see that they have different CRCs, but both are accepted as the same ROM by GoodSNES.
I still don't understand that. A binary diff of the ROMs is showing large parts that are equal, but obviously not everything. The size is the same, too.
Both do not run in emulators. They say bad checksum, but are listing C88748DB as the 'correct'(?) checkum for both ROMs. ZSNES tells me to deinterleave the ROMs.
So what's going on here? Please help me understand that.

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #19 on: June 26, 2017 - 15:17:19 »
Hi Zandro,

I have read and tested even more, but I still don't fully understand.
I now see that the GoodTools do some magic to match the files even if they don't 'perfect' match with CRC or SHA-1.
How can I find out which ROMs are matching 'directly' or how did you do that? Is there any way to get the hashes out of GoodTools?

Many TIA for teaching me :)

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #20 on: June 26, 2017 - 18:26:07 »
Your realizations were my own 5 years ago, and my findings are what I made this topic for.  The first post already mentions the common steps, the problems of trying to go further, and that I will not expose the hashes out of respect of the given license.  The GoodTools alleviates the need to learn about a bunch of formats (though they're basically the same). The average user can just use the tools without having to get messy with the roms.  For anyone else, it's about asking the right questions.

By the way, I still could not determine the matching properties for the Dennis rom.  That's why I asked here.
« Last Edit: June 26, 2017 - 18:36:03 by Zandro »

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #21 on: June 27, 2017 - 09:49:20 »
[You are not allowed to view links] Register or Login
... and that I will not expose the hashes out of respect of the given license.
I really respect this, too.
[You are not allowed to view links] Register or Login
For anyone else, it's about asking the right questions.
I see, let me try. I'm really willing to learn. :)
So what have I done:
I started with Good7800. A deep scan reveals various numbers. It seems the leading one is a CRC and the trailing one the SHA-1.
What drives me mad here: Whatever I try, I cannot get my files to match those values, neither with removing the header nor with setting it to 0.
So I tried with GoodSNES and Dennis. A deep scan gives me the following output for the 2 different files:
"(97C}6d97 87c15591 Dennis (E) [hI]|72a945db b139c6c9 7929cb91 5dd24a2d 5c0a3edc",
"(97C}6d97 c88748db Dennis (E) [hI]|33c0f261 9e5e0ccb 2ab061f6 d7d905dc d3f6dfe2",
Voila, here is the CRC, but with another leading number (not existing in Good7800), I don't understand.
If I understand you correctly, you want to find out, how to 'convert' one to the other to understand the matching algorithm here. Is that right?
Both ROMs are matching, so I diffed them, but I don't see the magic here yet.
Maybe you could point me in the right direction, so I can help.
Sorry for my bad english, I'm not a native speaker.

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #22 on: June 27, 2017 - 20:59:16 »
Quote
I started with Good7800. A deep scan reveals various numbers. It seems the leading one is a CRC and the trailing one the SHA-1.

Old, simple 7800 went out the window last year.  From Good.doc:
Quote from: Notes for Good7800 V3.28
No .bin files anymore. I am converting Good7800 into
something more like GoodNES, with a 'fix' option and
all files must have a .a78 header. Once I'm happy
with the headers I am fixing to work with most (if not all)
emulators, I'll issue a new version that can add the
header to a .bin. If you own a CC2 and don't already
know how to strip a header to go from .a78->.bin you need
help!

None of my tests yielded exact hash matches with the newer version, so I gave up trying.  As he says, improving compatibility is more of a focus right now, so having verified headers and a future tool to generate them should be nice.  It's easy to go from headered back to bin, but not the other way.

Also, I do not reference the deep scan hashes.  They are of the roms scanned, but I use an external tool (hashtab) to check their CRC/SHA-1 already. It doesn't matter what is used, but you cannot learn the tool's internal hashes with deep scan. 

Quote
you want to find out, how to 'convert' one to the other to understand the matching algorithm here

Not vitally by any means, I am only passively curious.  If I don't know how, I don't believe I can teach you how either.  Independent exploration derives independent results.  If I lead you down the wrong road, you could get just as lost as me.  But, here is what I see:

Unfortunately, I don't know what {97C}6d97 is.  Deep scan's details are not explained in his docs, and I couldn't find any info with Google.  I should ask Cowering.

The rom contains an intro using some relocation pointer.  The intro itself is stored at the end identically in both roms, so that data is irrelevant in attempts to convert between either rom.  North of the 1MB boundary is totally different, so it's not just an internal header or checksum difference.

ucon64 gives an additional search checksum, but only in the "real" [hI]. My odd one does not.  According to ucon64, the real one is interleaved, and mine is not.  This is surely an oddity, as "pure" hashes are supposed to be of de-interleaved roms.

RNC headers typically indicate [You are not allowed to view links] Register or Login, but that data type is in both variants, just rearranged in the rom, due to the block interleave.  The compression is present in the verified dump, so it is not hacked in by the scene and does not need to be reversed.  Dead end.

I'm out of time for this now.
« Last Edit: June 27, 2017 - 21:16:55 by Zandro »

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #23 on: June 28, 2017 - 16:15:43 »
I suppose I found the problem.  As ucon64 de-interleaves the rom, it removes the 16KB intro from the padded region.  Putting the intro back on results in a match with my "odd" rom.

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #24 on: June 28, 2017 - 16:48:10 »
Great to read that and thanks for the tip :)
I'm still busy investigating how to find out wether a rom matches 'directly', but with very few success.

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #25 on: July 03, 2017 - 09:44:03 »
Ok, I don't get it.
Is there any way to get the hashes out of the GoodTools? Or is it even possible to recognize a 'direct' match without having the hashes?

Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #26 on: July 03, 2017 - 20:49:23 »
The hashes are protected, but recognition can still be done intuitively given what I shared.  But don't obsess.  For 7800, I just have to wait for the next version to see what's expected.

Offline yeahimdukenukem

  • Full Member
  • ***
  • Posts: 183
Re: Set Sprucing
« Reply #27 on: July 03, 2017 - 22:50:00 »
Today I was really frustrated in finding out how to recognize a direct match.
So I decided to just let it be and test various ROMs in various emulators if they are working nevertheless.
And the frustration increased as really about a third of the tested ROMs did not work although the ones from No-Intro did.
Same with the Dennis ROM, mine did not work, yours did. I was just on the verge of deleting the GoodSets, so I can sleep again...
But I'm not the one giving up.
The hashes are not needed if only the GoodTools would tell you that a ROM directly matches or not.
Ok, for 7800 we have to wait, but you mean for the other sets the recognition can be done, even without the hashes. Maybe it's because of my bad english, but I really must be missing something here, right?

Offline attractivo

  • Hero Member
  • *****
  • Posts: 586
Re: Set Sprucing
« Reply #28 on: July 03, 2017 - 23:03:47 »
you could check the deduplication project dats, as they contain goodset dats which include only best hashes for each set. and also seperate dats for known variant roms aka duplicates.

though the goodsets are deduped with nointro and zandros smw hacks. if you also collect them first, you are nothing missing besides spc and psid sets for which no besthash dats were generated.


Offline Zandro

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3482
  • Bézier Extraordinaire
Re: Set Sprucing
« Reply #29 on: July 03, 2017 - 23:31:55 »
Quote
So I decided to just let it be and test various ROMs in various emulators if they are working nevertheless.
And the frustration increased as really about a third of the tested ROMs did not work although the ones from No-Intro did.

Specifically which roms?  NES roms generally need header information for many emulators, but they are NOT included within the tool's hashes.  For some sets, removing or altering headers breaks rom playability.  This is why I continuously ask others to NOT try to match them perfectly.

Other than headers, pay attention to the tags: [a], [b], [f], [o] (altered, bad, copier-fix, overdump).  These often don't work correctly in emulators, nevertheless they are included in the GoodSets because that's the whole point of GoodTools: To identify and label roms to give you knowledge of the state of them.
« Last Edit: July 03, 2017 - 23:34:05 by Zandro »