Golden Sun Hacking Community
01, October, 2020, 02:49:41 PM *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: The last working version of the classic TLA Editor is Teawater's unofficial "mapeditor" draft.
 
  Home   Forum   DC Wiki Help Search Calendar Downloads Login Register  
Pages: [1] 2  All   Go Down
  Print  
Author Topic: Compression Formats  (Read 12999 times)
0 Members and 1 Guest are viewing this topic.
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« on: 20, February, 2015, 10:29:34 PM »

Unlike some games that have only one or two compression formats, Golden Sun has a bunch of them.

Format 0 = - Used by World Map Tilesets, Regular Map Palettes, Map Code
My bit code table has 1 bit!
0 = Seems to match the distance-length stuff described in Format 2.
1 = 8-bit constant

Format 1 = Distance-length (LZ77 variant?) - Used by World Map Palettes, and maybe other various things.

8 flags for eight entries.
If 0: 8-bit Constant
If 1: Distance length. (aaaa.aaaa.bbbb.cccc = b is most significant bits of distance, a is least significant bits of distance, and c is length.)
If length and distance are both 0: End.
If length is 0, but distance is not, get another byte for length from compressed stream and add 0x10 to it. (Distance-length extended.)


Format 2 = Complex format... (See code at 0x08012B70) - Used by Regular Map Tilesets

The first part is distance-length data. (The bit furthest to the right is 0.) ; The number I put at the right is the length.  See next section for distance stuff.

---.----.----.--00 = 2
---.----.----.-010 = 3
---.----.----.0110 = 4
---.----.---0.1110 = 5
---.----.-001.1110 = 6
---.----.-101.1110 = 7
---.----.xx11.1110 = 8-10
xxx.xxxx.0011.1110 = 11-137
000.0000.0011.1110 = End



Each of these above (except for "End") have distance data attached to the left end as well. (Not shown in the bit code above, though. Since it might make it look more messy?)

a.aaaa.aaaa.aaa0 = a represents distance + 0x20. (The number of bits (a's) can vary here... likely depending on stream length, but it stops at a max of 12-bits.)
-.----.--aa.aaa1 = a represents distance (Always 5-bits.)


FEDCBA9876543210 part: (See Icon decompressor for the basics.)
x = For left half of byte.
y = For right half of byte. (Should be done first.)

---.--xx.yy11 = &3 ; Accesses the first 4 slots.
--x.xxyy.y101 = &7 ; Accesses the first 8 slots.
xxx.xyyy.y001 = &F ; Accesses all slots.

.

Icon decompressor =  This is a simple format that uses the FEDCBA9876543210 list like Format 2 does. When you get a number from this list, that number is moved to slot 0, and the rest of the numbers are pushed over.

Bytes can only be decompressed to 00-0F with this format.

The following table I made is formatted like this:
Binary code on left, The bits are varied in length from 1-10 depending on slot number, so that the more freqently accessed used numbers which are put in the first slots take fewer bits to reference.
Address for the code that takes the value from the value list, and puts it at the head of the list.
Slot number at end.

--.----.---0 = 08038B5C = 0
--.----.-001 = 08038d74 = 1
--.----.0101 = 08038d4c = 2
--.----.1101 = 08038d60 = 3
--.----.0011 = 08038d18 = 4
--.----.1011 = 08038d04 = 5
--.----.0111 = 08038bb4 = 6
--.--00.1111 = 08038be8 = 7
--.--10.1111 = 08038bf4 = 8
--.--01.1111 = 08038c10 = 9
--.0011.1111 = 08038c44 = A
--.1011.1111 = 08038c58 = B
--.0111.1111 = 08038c7c = C
00.1111.1111 = 08038cb0 = D
10.1111.1111 = 08038cc4 = E
01.1111.1111 = 08038ce8 = F
11.1111.1111 = End

FEDCBA9876543210

For example, if you were getting from slot 8, since the number 8 is found at slot 8, you take that number and move it to the beginning like so:

FEDCBA9765432108

If you need the number 1, you will now need to get from slot 2... which will move the number to the head of the list, again, as so:

FEDCBA9765432081

And so forth.


Battle Background = Documented in another topic ( http://forum.goldensunhacking.net/index.php?topic=2012.msg37023#msg37023 )

This format is about add/subtracting (relative difference) from last value to get the next value. This format is meant for non-tiled battle backgrounds where each decompressed byte ranges from 0x60 to 0xDF. (128 possible values because of 128 colors in the battle background palette.)

--.----.--00 = + 0
--.----.x001 = + 1,2
--.xxxx.y010 = +/- 11-26 (0x1A in hex) ; y=sign
--.--xx.x011 = + 3-10
--.----.x101 = - 1,2
xx.xxxx.x110 = = value (Relative from 0x60, aka, beginning of background's palette.)
--.--xx.x111 = - 3-10


Palette Menu graphics compression = To be documented here...

16-bits for 16-flags.
If 0: 16-bit constant
If 1: 16-bit distance-length ; 0x0000 = End? ; 0xFFE0 = Distance ; 0x001F = Length

Text compression = Huffman variant... - Each character has a tree of characters that can come next. (This is why if letter combos don't exist in the original text data, you won't be able to compress unless you rewrite these trees.) The trees are arranged by node frequency the same way huffman is normally done. (Google to learn more.) The start of a sentence starts with the tree for char 00, and char 00 also end the sentence.

08060C30 = Pointer list containing character data...
There are only two pointers. (This is one entry.) More entries may be supported, though.
Second pointer list the relative location from the first pointer... Basically each relative pointer is for each character that you last decompressed... And brings you to where the list of characters that can come after (12-bit table which appears before the calculated address.) The calculated address points to the character's tree. If you count all the ones (in binary), it should ammount to the same number of characters in the character's character list.
The characters in the tree are ordered by frequency of the same length of bits. (I mean the nodes have frequency values as well... not just the symbols...)  - You can find some documentation on building trees for Huffman compression.... It's quite likely the same thing.

080A9F54 = String Table ; Each entry is two pointers. Each being used for 0x100 strings at a time. First pointer is compressed text data, second pointer is length of each string. If length is 255+, putting in a 0xFF will also read the next byte and add to the 255.... so no 255 char limit here. Although, there might be a ~512 limit in the parser function, though.)

Demonstration:  Let us compressed the string "Teawater"

Char 0x00's list has just a "T" because this is a single line. (The only time a char can come after char 0x00.)
Char "T" 's list has an "e"... (frequency 2, but since there's not other letters to compare it to, amounts the same as above.)
Char "e" 's list has a "a" and an "r" .... equal frequency. (Forces the "text data" to now need one bit of data.)
... etc. Maybe I fix this up better a bit later / dunno, though.



Sprite Compression Formats:
Type 0 = No compression??
Type 1 = Uses Main Format 0/2.
Type 2 = ?
Type 3 = LZ77 variant - Looks like the sections are divided up. (Bytes used in distance length / Constants / Flags and Distance Length pairs. ; If value at base address is 0000, read as though it is uncompressed. ; For the compression, distance-length pair should be 0000 to end. ; Distance length as a 16-bit: xyyy ; x=Length, y=Distance
« Last Edit: 24, June, 2017, 03:13:57 PM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
Misery
Bad Luck

Great Member
***

Coins: 20
Offline Offline

Gender: Male
Clan Position: Mercury Hack Leader
Posts: 717

« Reply #1 on: 21, February, 2015, 03:01:55 AM »

Wow, I had completely forgotten you had cracked the battle background decompression already. I think it's really neat that you're looking into and documenting these formats.

Icon decompressor =  This is a simple format that uses the FEDCBA9876543210 list like Format 2 does. When you get a number from this list, that number is moved to slot 0, and the rest of the numbers are pushed over.

Bytes can only be decompressed to 00-0F with this format.
So this is used for things like equipment and psynergy icons?

Quote
Binary code on left, The bits are varied in length from 1-10 depending on slot number, so that the more currently accessed used numbers which are put in the first slots take fewer bits to reference.
I assume you mean frequently?
So each slot holds a palette index number from 0-F of the global palette?

How is the slot order determined when there's an equal amount of pixels of different colors?

When is the end "bit" (%11.1111.1111) used? Is it at the end of each tile? Or the end of a full table of graphics data?

Sorry if these questions are stupid/annoying. If I can fully understand how this decompression works, I'd like to try writing a tool for converting other images into this format. Possibly even a full icon editor, but I'm not sure if I'd be able to do that.
Logged
View Profile
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #2 on: 21, February, 2015, 03:25:18 AM »

Quote
So this is used for things like equipment and psynergy icons?
I recorded this function as being for icons a long time ago, but I only recently figured the format out not long ago. I think it may apply to Item, Psynergy, and Status icons? (Basically 16x16 icons) I will need to check this out again, though, as well as to see if anything else uses this.

Quote
I assume you mean frequently?
Yep, frequently sounds like the right word. (Or is "recently" a better word?)

Quote
So each slot holds a palette index number from 0-F of the global palette?
Yeah...

Quote
How is the slot order determined when there's an equal amount of pixels of different colors?
I'm a little confused at what you mean?
At the start of the stream, it should always start with this order: FEDCBA9876543210 regardless of the data itself. For every entry that gets decompressed, the value moves to the head (slot 0) and pushes the rest down..... (That way, if you need the same color/palette index again, it just takes one more bit to get it... ("0"))

Quote
When is the end "bit" (%11.1111.1111) used? Is it at the end of each tile? Or the end of a full table of graphics data?
The latter.  "End" goes at the end of the compressed file to say that there is no more data to be decompressed.



Format 2 may follow a similar concept to the icon decompressor, but I think it assigns to both sides of the byte instead of one... and the bit format is different.
« Last Edit: 21, February, 2015, 05:07:16 AM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
Misery
Bad Luck

Great Member
***

Coins: 20
Offline Offline

Gender: Male
Clan Position: Mercury Hack Leader
Posts: 717

« Reply #3 on: 21, February, 2015, 07:43:44 AM »

Quote
How is the slot order determined when there's an equal amount of pixels of different colors?
I'm a little confused at what you mean?
At the start of the stream, it should always start with this order: FEDCBA9876543210 regardless of the data itself. For every entry that gets decompressed, the value moves to the head (slot 0) and pushes the rest down..... (That way, if you need the same color/palette index again, it just takes one more bit to get it... ("0"))
Ah, it was just me jumping to conclusions... basically I thought the order was a constant thing based on the most-to-least commonly used index. Yeah, that's completely different from how you explained it, so just don't mind that. And I guess 'frequently' still works, but not in the sense I imagined.

So as I understand it now, every time the next entry is different from the most recent entry, that new entry gets the shortest bit length assigned to it after it has been interpreted, pushing all the others 'left' one step. Sorry if that's a confusing way to phrase it, lol. I think I get it now at least.
Logged
View Profile
Salanewt
His Sexiness
Global Moderator

Oh yeah, baby!

Prodigy
*

Coins: 203
Offline Offline

I am: A part of the organization of Cool Cats, but more of a dog person in reality.
Emblems: Have a nice day.
Posts: 4605

« Reply #4 on: 22, February, 2015, 04:34:08 PM »

Oh sweet, nice topic! I'm just going to try clarifying on a couple points for the icon decompression stuff for those who may struggle with following it though (myself included). Sprite/icon palette below for ease of clarification, though I have to be honest that your post has me a little confused in a couple areas. Should be able to figure it out after reading through it if all goes well.



Quote
--.----.---0 = 08038B5C = 0
--.----.-001 = 08038d74 = 1
--.----.0101 = 08038d4c = 2
--.----.1101 = 08038d60 = 3
--.----.0011 = 08038d18 = 4
--.----.1011 = 08038d04 = 5
--.----.0111 = 08038bb4 = 6
--.--00.1111 = 08038be8 = 7
--.--10.1111 = 08038bf4 = 8
--.--01.1111 = 08038c10 = 9
--.0011.1111 = 08038c44 = A
--.1011.1111 = 08038c58 = B
--.0111.1111 = 08038c7c = C
00.1111.1111 = 08038cb0 = D
10.1111.1111 = 08038cc4 = E
01.1111.1111 = 08038ce8 = F
11.1111.1111 = End

So basically, an icon can use any of the colours in the first row, and it reads them using binary format. Additionally, it looks like any %0s that come after the last %1 will push everything else to the right by a pixel per %0 (defaults to transparency if no colour is assigned), but I may be wrong on that part. So for example...

x3 = %11, meaning that it will be yellow (fifth colour when including transparency)
x5 = %101, meaning that it will be red (second colour)
xC = %1100, meaning that it will be yellow, and everything is moved/shifted to the right by two pixels, which basically extends the length of the previous colour (I think?)
x1F = %11111, meaning pink
x20 = %100000, meaning black, and everything is shifted by five pixels

And looking at the post again, all remaining pixels will be the previously used colour until you use another one. It looks like you're saying that one can reserve colours for later so they don't have to be recalled every time they are needed, but I'm not really sure I understand how that is done from your explanation.

I don't know, does it sound like I'm getting this right so far?
Logged

Oh yeah baby, £ me harder.

Fusion is just a cheap tactic to make weak Adepts stronger.

Yoshi's Lighthouse is a hacking website in progress. Why not check it out if you like Yoshi or the Mario & Luigi games?
View Profile WWW
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #5 on: 22, February, 2015, 11:05:26 PM »

Oh, I see, you're thinking about it directly from the colors point of view.  Sounds like a good way to put it for new people. (At least, before going into the bit structure and stuff. But that stuff also helps for the technical people.)

Once you use a color, it is put at the head  (slot 0)  of the 0x64 bit number (The number is really divided into two 32-bits) ... and everything else is pushed over to fill the gap that was opened. = This is the basis of that compression, since slots at the head take fewer bits to reference then at the tail.  And my guess is that if you're using one color, the chance that you'll use it again, even for just the very next pixel is probably more likely than other colors? (Well, for horizontal lines, and fillers.)
Pretty much, I just did it to explain that the 0xFEDCBA9876543210 holds indexes to palette, and the values there are rearranged. In the compression, holds variable length bit codes that are used for getting a slot in this number. (Slot 0x0 is the right side, and slot 0xF is the left side... in this 0xFEDCBA9876543210 number.)

Quote
Additionally, it looks like any %0s that come after the last %1 will push everything else to the right by a pixel per %0 (defaults to transparency if no colour is assigned), but I may be wrong on that part. So for example...
Not sure I understand this? ; Either way, you'd get the slot for where the transparent pixel is.

Quote
x3 = %11, meaning that it will be yellow (fifth colour when including transparency)
x5 = %101, meaning that it will be red (second colour)
xC = %1100, meaning that it will be yellow, and everything is moved/shifted to the right by two pixels, which basically extends the length of the previous colour (I think?)
x1F = %11111, meaning pink
x20 = %100000, meaning black, and everything is shifted by five pixels
1: Correct, but you still need the 0s attached: %0011, otherwise, it could be more confusing. Think of it like (val & 0xF = 0x3), (Although, the asm code itself shifts one bit at a time, and does conditional branching each bit that get shifted out.)
2: If this is the first grab, correct, if this is the second grab in your current list, no. (The fifth color would have moved to the first slot, and the rest pushed down to fill the gap.)
3: Wait? 1 color grab, and shift by 2 pixels? I'm not sure I follow? %1100 looks like three colors, though. Slot 0, Slot 0, and Slot 4 if you add on the 0s to the left.
5: It just looks like you are putting several bit codes into one....

Quote
It looks like you're saying that one can reserve colours for later so they don't have to be recalled every time they are needed, but I'm not really sure I understand how that is done from your explanation.
How do you mean? All that happens when you get a color, is the color moves to the head of the (64-bit), and the rest get pushed down. (Both to make room for this color being at the head, but also to fill the gap where this color was previously....... and this happens with all bit codes.)

On the topic of using the % thing, I guess we use these?:
%, b = bits
0x, h = Hex
0o = Octal

---

Even though I know this compression format ~99%(Because I haven't looked into compressing, yet, just decompressing.), it's probably hard to explain it in a user-friendly fashion, so sorry for any inconveniences.
« Last Edit: 22, February, 2015, 11:27:10 PM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
Salanewt
His Sexiness
Global Moderator

Oh yeah, baby!

Prodigy
*

Coins: 203
Offline Offline

I am: A part of the organization of Cool Cats, but more of a dog person in reality.
Emblems: Have a nice day.
Posts: 4605

« Reply #6 on: 22, February, 2015, 11:48:59 PM »

Aaah, okay. So the colours get rearranged as you use them?

This makes me wonder something though. How should I view the code if I want to try my hand at making one manually, 8-bit? I'm still not sure I understand how colours are shifted to slots though, and it kind of looks like 4-bit didn't work too well for me either. All I know is that I would certainly like to try this out for a bit at some point, and that I think I may be starting to get it.
Logged

Oh yeah baby, £ me harder.

Fusion is just a cheap tactic to make weak Adepts stronger.

Yoshi's Lighthouse is a hacking website in progress. Why not check it out if you like Yoshi or the Mario & Luigi games?
View Profile WWW
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #7 on: 23, February, 2015, 12:13:15 AM »

Quote
Aaah, okay. So the colours get rearranged as you use them?
Yes. The 64-bit value represents all 16 colors.

I was hoping that this part of my first post would clear up any confusion left behind:

Quote
FEDCBA9876543210

For example, if you were getting from slot 8, since the number 8 is found at slot 8, you take that number and move it to the beginning like so:

FEDCBA9765432108

If you need the number 1, you will now need to get from slot 2... which will move the number to the head of the list, again, as so:

FEDCBA9765432081

And so forth.
That's where I show how they're effectively moved around. And I also mentioned the 0-F in the bit code table was for slot/position number in that 64-bit number.... by this quote, anyway
Quote
Slot number at end.

---

@8-bit: Like Format 2?
« Last Edit: 23, February, 2015, 12:16:29 AM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
Salanewt
His Sexiness
Global Moderator

Oh yeah, baby!

Prodigy
*

Coins: 203
Offline Offline

I am: A part of the organization of Cool Cats, but more of a dog person in reality.
Emblems: Have a nice day.
Posts: 4605

« Reply #8 on: 23, February, 2015, 12:48:47 AM »

Ah, okay! Yeah, I just wasn't sure because the only time you actually used either the word "color" or "palette" was in the battle background portion, and you just referred to them as slots before then. It wasn't really clear whether you were talking about that or something else, though the x00-x0F reference and Misery asking about the global palette did clear that up for me. Unfortunately, that quote just made things a tad more confusing because it didn't really specify how the colours are referenced, just how they are rearranged. It was also a bit confusing with the use of "frequently/currently," since neither of those really imply that they are arranged as you reference them. All that said, I feel rather confident that I understand it better now.

8-bit: Oh, so that's how I read it? Sweet, thanks!
Logged

Oh yeah baby, £ me harder.

Fusion is just a cheap tactic to make weak Adepts stronger.

Yoshi's Lighthouse is a hacking website in progress. Why not check it out if you like Yoshi or the Mario & Luigi games?
View Profile WWW
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #9 on: 23, February, 2015, 01:04:20 AM »

Quote
Ah, okay! Yeah, I just wasn't sure because the only time you actually used either the word "color" or "palette" was in the battle background portion, and you just referred to them as slots before then. It wasn't really clear whether you were talking about that or something else, though the x00-x0F reference and Misery asking about the global palette did clear that up for me. Unfortunately, that quote just made things a tad more confusing because it didn't really specify how the colours are referenced, just how they are rearranged. It was also a bit confusing with the use of "frequently/currently," since neither of those really imply that they are arranged as you reference them. All that said, I feel rather confident that I understand it better now.
Probably should be revised/rewritten?
Well, frequent in the sense that if you keep moving something to the top of the list, it may be rarely put at the tail. Either way, do you think "recently" works best?

Quote
8-bit: Oh, so that's how I read it? Sweet, thanks!
I was assuming maybe you wanted one like this that decompresses to values of 00-FF. (Although, Format 2 splits it up, still... into the left and right sides.)
I probably misunderstood, though...
Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
Salanewt
His Sexiness
Global Moderator

Oh yeah, baby!

Prodigy
*

Coins: 203
Offline Offline

I am: A part of the organization of Cool Cats, but more of a dog person in reality.
Emblems: Have a nice day.
Posts: 4605

« Reply #10 on: 23, February, 2015, 04:14:03 AM »

Perhaps, yeah. I would suggest changing this:

Quote
so that the more freqently accessed used numbers which are put in the first slots take fewer bits to reference.

Into something more like this:

Quote
rearranging the colours in a way that makes them easier to reference later in the code.

Or something like that. That way you could perhaps clarify on how this is done later in that section. Just an idea though, since I think I understand what the whole section means now.


Ah, that's what you meant! What I am hoping to attempt is to make a few custom icons for some of the spells I am incorporating into Disco Sun, as I figure that I could potentially do this now so I can get it out of the way and not have to worry about spell icons for a while. While I understand that doing this manually could be a painful experience, I am definitely willing to put the effort in; besides, I generally do assembly without assemblers or anything like that.
Logged

Oh yeah baby, £ me harder.

Fusion is just a cheap tactic to make weak Adepts stronger.

Yoshi's Lighthouse is a hacking website in progress. Why not check it out if you like Yoshi or the Mario & Luigi games?
View Profile WWW
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #11 on: 23, February, 2015, 06:29:41 AM »

Quote
rearranging the colours in a way that makes them easier to reference later in the code.
I think that's still too vague... Since I doubt that anything needs to be made "easier" to reference.... Instead, just less space/bits to reference. ; (And some people may even object and say it's harder... since it would take humans a bit more time than uncompressed.)

Quote
While I understand that doing this manually could be a painful experience, I am definitely willing to put the effort in; besides, I generally do assembly without assemblers or anything like that.
Or I, you, or somene else could code a converter.... Waaayyy easier then coding an assembler editor in comparison.
You could also consider decompressing the icons and having them in that format. (AKA: Get rid of the icon decompressor altogether.)

@Text compression: Still studying it, but Terraria had me side-tracked....

08060C30 = Character data? - Two pointers, it's actually a list of one entry. (One entry here is two pointers.) I think the code supports more entries, but they're not needed.

The second pointer has a list of relative addresses for the first pointer... And there are two segments of data that can be found through the first pointer... for each relative address in the second pointer...
One segment comes before the address (The one after relative is calculated in), (12-bits per entry), and the other segment starts at the address. (Variable-length bits...)
I think the first segment is a char list? ....
Is the second segment some type of tree?  I have a feeling it is! (Tree dimensions? And the characters get assigned to them in order?)
Might need to confirm this stuff further, though...

@Second segment
My current theory is that the tree goes as deep as how many bits of 0 you have until you reach a 1... (And repeats for the tree's branches.) ... And in the text data, you put how deep you want to go, as well into each branch... and so forth. Until you get to the end of a branch, which would be your letter.

Are there as many bits that are "1" as there are number of characters branching from a character? Looks like it! ... So 0 = branch, 1 = End of branch.....

I may have to actually use one of the character's tree data in an example when the tut comes? (I'm guessing I /might not/ need to use an text example string from GS, though?) - It'd be quite a lot of work to manually compress your own text strings, though, if you have to go and look up all the trees for each character you use. -- On second thought, the first character (Non-character?/Starter tree when you decompress the first letter of a string???) has 82 possible character connections.... so that might be a bit much. I could maybe consider Char 0x12? Or not, since it seems to have only one character in its list...?
« Last Edit: 23, February, 2015, 01:10:03 PM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
approxies
Guest
« Reply #12 on: 14, November, 2015, 06:25:31 PM »

Hello, everyone. Being interested in compression theme for GS, I've made some tools for graphics packing/unpacking:
What Fox calls Format 1 and Format 2 (unpacker only)

Could someone please update me if graphics compression problem is solved already and there's no need in such tools in community?
Logged
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #13 on: 14, November, 2015, 11:44:09 PM »

Good... Did I forget to link this to pastebin? You can find some decompression routines I wrote here... http://pastebin.com/pzhYZu7G (C#) (I forget if I made fixes to compress16 for the Icon Compressor since then, but I'll need to double check.), but you can also find some stuff in Atrius's editor source.

Porting to Python and other languages could be interesting as well.. but hmm..


--

Huh... Seems I'm still not familiar with Haskel... but haven't taken a lot of effort to learn it either. - At least byte-level code is understandable to me. (Whether it be VB, C#, GML, etc.) :) (I feel like it was a stupid mistake a long time ago when I chose to learn VB.NET... there are so many other better languages for my taste... maybe even Haskel that I could have went for... But either way, at least I chose a language at all.)
« Last Edit: 15, November, 2015, 06:47:49 AM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
approxies
Guest
« Reply #14 on: 15, November, 2015, 10:30:03 AM »

You did OK with .NET platform decision. Haskell has huge issues with GUI applications development and it's not popular, so a small amount of people will understand the code. But it's clean code and rapid development loop is fine when I need to write some small CLI tool for my purposes.
So, bottom line, both compression and decompression for text and graphics are supported by Atrius's editor and there's no need for separate tool?
Could you also update me where can I find GS ROM mapping for more compressed graphics offsets?
Logged
Daddy Poi's Oily Gorillas
Fox McCloud, the Hacking Doctor
Mercury Clan

Prodigy
*

Coins: 1002
Offline Offline

I am: certainly not a Gallant!
Clan Position: Head Gallant
Posts: 2491

« Reply #15 on: 16, November, 2015, 02:49:45 AM »

Well, it didn't fully support Text Compression, as Atrius's editor never rewrote the Char tables... It also didn't decompress battle backgrounds... etc. But the point was that it did support decompressoin/compression for some things...

I think most images are in three/four sections.
1.) The sprites tble: (0x08300000 in GS2)
2.) The Master File Table.. (0x08320000 in GS1, 0x08680000 in GS2; Yep, I have both of them memorized.)
3.) One of the code files has compressed icons.
4.) Oh, and there might be some things compressed in another code file files. (Like the palette menu graphics, I think..) but other than that, I think most graphics in those files are uncompressed(?) - Not many uncompressed images, though... but the red crystal grpahic (for convos), and that pink/purple conical selection thing used to select an enemy maybe.. some |> arrows used in debug menus....  etc. That kind of thing.)
« Last Edit: 16, November, 2015, 03:01:22 AM by Fox » Logged

Golden Sun Docs: Broken Seal - The Lost Age - Dark Dawn | Mario Sports Docs: Mario Golf & Mario Tennis | Misc. Docs
Refer to Yoshi's Lighthouse for any M&L hacking needs...

Sometimes I like to compare apples to oranges. (Figuratively) ... They are both fruits, but which one would you eat more? (If taken literally, I'd probably choose apples.)
Maybe it is over-analyzing, but it doesn't mean the information is useless.


The only GS Discord servers with significance are:
Golden Sun Hacking Community
GS Speedrunning
/r/Golden Sun
GS United Nations
Temple of Kraden

Can you believe how small the Golden Sun Community is?

2+2=5 Don't believe me? Those are rounded decimal numbers. Take that, flat earth theorists! :)
View Profile
approxies
Guest
« Reply #16 on: 16, November, 2015, 06:39:09 PM »

OK, thanks for getting me inputs, I will investigate further. I guess, pointer for the chunk of graphics, I'm going to translate (GO and 1st match glyphs in arena) is hardcoded.
Logged
approxies
Guest
« Reply #17 on: 11, February, 2016, 05:06:33 PM »

An update for my set of compression tools of Golden Sun.
Now it supports decoding/encoding for text, and two LZSS schemes for graphics, described in OP post.
Logged

Novice Member
*

Coins: 0
Offline Offline

Posts: 13

« Reply #18 on: 10, March, 2016, 12:51:56 PM »

Hi approxies, Fox and Lord Squirtle, I am currently looking for a way to encode my Polish translation of The Lost Age in a different way than using Atrius' Editor, as it makes the output file much bigger and is now incompatible with The Balance Age  Sweat Drop
Needles to say that this thread caught my interest. Approxies, would you mind making a little tutorial to your compression tools for a non-programmer? I've played around with the tools a little bit, but can't really get them to work.
The haskell one returns "Could not find module 'Data.Bitstream.Lazy' (I just installed the GHC compiler).
Recommended translation scheme from the earlier version with .bat files works fine until the encoding (-e) part, where it shows "trees: Patch size is larger than specified. Aborted." I guess it is because my translation is larger than the original file (duh), so do I need to specify a new treeSize, Offset? How do I calculate this? :)
Here is the script, if you want to take a look.
Logged
View Profile

Excellent Member
****

Coins: 20
Offline Offline

Gender: Male
Posts: 907

« Reply #19 on: 10, March, 2016, 01:16:28 PM »

I'm not sure about it, but i think you can tweak gstlatoolkit itself and make it store the script later into the rom . that way you'd also gain back the compatibility with my patch.
Logged
View Profile
Pages: [1] 2  All   Go Up
  Print  
 
Jump to:  

Chatbox
31, August, 2020, 08:40:32 AM
Daddy Poi's Oily Gorillas: Being able to convert IPS files to text files to also supporting commenting, and convert back to IPS after changes.... sounds like a fun strategy? 
27, August, 2020, 10:37:30 PM
Salanewt: Hey, thanks! 
25, August, 2020, 12:59:22 AM
Foreclosure: Yo! Looking forward for this AI OVerhaul patch when it's finished :) :) 
19, August, 2020, 07:00:17 PM
Salanewt: That's awesome! I'm pretty good; been working on an AI Overhaul project and have made some nice progress in the past couple weeks; the goal for tonight is to add Deadbeard to GS2 and then I can move onto something new.
19, August, 2020, 01:14:56 PM
Awec: Ye not bad, just checked back in for updates on GS Reloaded really. Although now I'm looking back through some of my old forum posts, which is fun :D how are you?
19, August, 2020, 01:43:07 AM
Salanewt: It sure is! How have you been?
16, August, 2020, 09:40:57 AM
Awec: Yo nice to see this forum is still going
14, August, 2020, 09:06:44 PM
Salanewt: Hey! How goes it?
01, August, 2020, 01:31:53 PM
KyleRunner: Hello, people!
08, July, 2020, 08:57:11 PM
Salanewt: That awkward feeling when almost a quarter of a year goes by without anyone posting a message in this chatbox.
27, April, 2020, 07:54:10 PM
Daddy Poi's Oily Gorillas: I'm having a thought. So I know Atrius's editor is slow at saving sprites.... So I've been thinking.   I am considering another console program.  This time, for importing and exporting sprites/animations. Image files (for sprites), binary files (for animation data), and maybe binary or text files for Settings. Would certainly be cool if it works out. - Console programs are fun, when you can (at least temporarily) cheat out making a full interface editor for them. =P
25, April, 2020, 10:42:07 PM
Daddy Poi's Oily Gorillas: Tried to list those in order of importance, but anyway....
25, April, 2020, 10:37:59 PM
Daddy Poi's Oily Gorillas: GSHC, GS Speedrunning, /r/Golden Sun, GS United Nations, ToK.  Sixth one is border-line I'm not sure.
25, April, 2020, 08:00:21 PM
Daddy Poi's Oily Gorillas: (low as in <100)
25, April, 2020, 08:00:04 PM
Daddy Poi's Oily Gorillas: So while there are quite a few GS Communities out there. There are only up to maybe ~6 Discords worth even the consideration of joining, I think. Most of the rest of them have a fairly low member count. (Of which I can tell?)
14, April, 2020, 10:49:35 AM
Daddy Poi's Oily Gorillas: The Essential Age could be what it stands for?
13, April, 2020, 05:23:32 PM
Misery: Tea editor... that's a cute name for it
01, April, 2020, 02:42:48 AM
JupiterDjinn: Also my browser is sending two o f my messages srry.
01, April, 2020, 02:42:06 AM
JupiterDjinn: Just got the reply. Ok my question is: how can I get the tea editor on Android, because I already have an emulator. But no computer.
31, March, 2020, 07:18:29 PM
Salanewt: Sure thing!
31, March, 2020, 03:38:27 PM
JupiterDjinn: Can I have a question answered?
28, March, 2020, 12:50:13 AM
Salanewt: I want to shrink it down in size later, but yup!
26, March, 2020, 03:24:45 PM
Daddy Poi's Oily Gorillas: The news box is back? :D
19, March, 2020, 05:59:16 PM
Salanewt: Nice work hiding the searchbots Atrius!
19, March, 2020, 04:23:34 PM
Salanewt: I was wondering when I said that stuff! May need to work it out with Kain.
19, March, 2020, 07:03:05 AM
Daddy Poi's Oily Gorillas: (The simplest being to not accept char IDs pass 255, I think.)
19, March, 2020, 05:33:15 AM
Daddy Poi's Oily Gorillas: Heheh.... I'm just testing. You pretty much gave it away. Hahah. It seems to look fairly convincing, but I'm sure there's an easy solution to preventing it.
19, March, 2020, 05:28:50 AM
Atrius: Whelp, there it goes... Just what I didn't want.  Have fun with that I guess...   
19, March, 2020, 05:20:25 AM
Daddy Poi's Oily Gorillas: So like. Replace the "e" with another "e"? Alright. Gotcha.
19, March, 2020, 05:04:41 AM
Daddy Poi's Oily Gorillas: Good work.

Affiliates
Temple of Kraden Golden Sunrise
Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!
Page created in 0.09 seconds with 22 queries.