In Atrius's editor, how would I also be able to read a given map's tile data?

[Triyence]

I want to be able to separately construct a map from the game file in a similar manner to how the editor does it. Does anyone know how this can be done? I guess the main things I need to know are:

• Where is the map data stored in the .gba file?
• What are the details on any compression algorithms that store it?
• Once decompressed, is it similar to how the memory viewer in VBA works? Essentially just tile pointers?
  

Or maybe someone can help with how I can get, for a given map, starting at tile 0,0, the palette, tileid, and flips for each tile (on each layer). For example if I'm looking at a 32x32 tile map, I want all this information (palette, tileid, flips) for each layer of the map.

When a map is loaded in VBA, does the entire map tile data get put somewhere in memory?

[Daddy Poi's Oily Gorillas]

Where is the map data stored in the .gba file?

(Edit: For GBA files, I usually give ROM addresses prefixed with 08, since that's where it is in memory viewer.)

Generally as files listed in the MFT. (MFT is at 08680000 in GS2, for example.) However, there may be a couple needed tables that are relevant...

The primary table being right here: (GS2)
080F17A8 = Room Table (8 bytes per entry)
0649 = Map Code File Index
00   = Area (When this changes, Area flags x300-x3FF reset.) (Used in room names.)
01   = Whether you are in a town or not?  Retreat will not work with 02.  Usually 01 or 02.
0000 = Index to MFT indexes in 0802F380 table for Map/Palette/4 Tilesets data. ; The World Map may use a different table instead. (below)
0000 = ? ; Usually 0000 or 0001.


0802ED34 = World Map File Indexes (Grassland, Snow, Waterfall, Snowstorm (Prox area))
0000019A = Palette
0000019B = Animation data
0000019C = Terrain Tileset
0000019D = Tileset 1
0000019E = Tileset 2
0000019F = Tileset 3
000001A0 = Tileset 4
000001A1 = Animation Tileset 1
000001A2 = Animation Tileset 2

What are the details on any compression algorithms that store it?

I may have to double check which ones are using which ones, Should be one of Compression formats 0, 1, and/or 2 in the Compression Formats topic. Most are starting with a Format value (0-2) as the first byte, but this wasn't the case for World Map Tile Data, if I remember correctly. (Which used Format 0x01, basically.)

When a map is loaded in VBA, does the entire map tile data get put somewhere in memory?

Yep.



Map data after decompression AND re-arranging/etc. /in RAM: (Directly after decompression/before re-arranging... there would be three tables: one is for the first two bytes in a tile, and the other two are for the other two bytes in a tile. )
02010000 = Tilemap (Including graphics, event indexes, heightmap, etc.)(32-bit a tile)
000007FF = Graphic tile
00000800 = Unused? (MSB of graphic tile in GS1)
00003000 = If stepped on, heightmap index of PC (/NPC?) will change. (0=Does nothing? 1-3 are heightmaps 0-2.)
0000C000 = Priority? (Whether the graphic tile appears in front or not.) - TODO: See if this is 2-bit.
00FF0000 = Event Connector? (For doors as well.) (FF=Wall, FE=Gap for hopping?)
FF000000 = GS1: Height data; GS2: Special tilescript flags (below). (GS2's "Height data" is stored separately.)
01000000 = ? (Used where bridges are.)
02000000 = ? (Seen used in Jupiter Lighthouse near Hover pads/slopes. No idea what it could be.)
04000000 = Determines whether to use the first or second encounters index.
08000000 = Hit/damage inducing.
10000000 = ? (Not sure if used or not?)
20000000 = Slope down.
40000000 = Climbable (Ladder/Vine) (Use same value on tile above or below to activate.)
80000000 = Hop. (Can hop from this tile.)

02020000 = Tile data (Mini-tiles) (F000=Palette; 0C00=Flips; 03FF=Tile)

World Map data has it switched: (0x02010000 is the mini tiles (But it's one byte per tile instead of two... since all tiles have access to the full 256 color palette, and flips don't occur.), 0x02020000-0x02021FFF is the map data (Only the section in which you are in is loaded b/c it's so huge.)... which may contain different data... (For the last byte in a tile, at least.)

Don't forget I also have coded this in my C# editor as well...  Maybe that could be useful. (Including GS1/GS2 world maps... (Some minor differences between them.)...It's possible to get an out of memory error b/c of the World Maps, though. (Esp. for the GS2 one.)... But with enough RAM, they should be able to load. - Forgot to delete/comment some test code that I had that may modify the map's display in GS1's World Map based on other data in the map data.... but I suppose it can be ignored.



---
@World Map:  Oh yes, the game also lists each map file index in a ram table... (Each set of 16x16 tiles (256 tiles) is a map file.) Not something I did with my editor of course, since it wasn't really needed.... Basically, those indexes can be changed to load other map files of the map instead...  That's kind of how the broken bridge south of Vault worked, actually....


---
Attachment includes my source code. In case it wasn't obvious, this is suppose to be an open-source editor. It seems I'm the only one coding it, but it is encouraged/open for others to code in it if they find that helpful to the Hacking Community. (Etc.)

[Triyence]

Thanks Fox. That source code is indeed useful. If I wanted to find the spot in your code that is retrieving the palette, tile id and flags for a given tile, where would I find that? The code is quite hard to read with all the magic numbers. Looks like it might be in decompTileData?

Also noticed that there is no separation of the map layers, do certain ranges in the stiles correspond to certain layers?

Edit

Playing around with the stiles variables but I'm not seeing where you correlate a read tile to given position on the map. I.e. where do you figure out that the tile at 0,2 in map 4 should be tile Id 774, palette 0?

Here are the first few tileids of stiles for map 4:

Code Select Expand

0772 0773 0804 0805 0774 0775 0806 0807 0902
0903 0934 0935 0900 0901 0932 0933 0705 0773
0737 0805 0705 0835 0737 0867 0836 0837 0868
0869 0838 0839 0870 0871 0836 0839 0868 0871
0840 0714 0872 0746 0000 0000 0000 0000 0000
0000 0000 0130 0376 0377 0408 0409 0378 0379
0410 0411 1024 0000 1024 0000 0000 0000 0000
0258 0000 0000 0259 0000 0512 0545 0545 0512
0513 0513 0545 0545 0512 0513 0512 0545 0609
0608 0513 0512 0545 0545 0577 0577 0512 0545
0576 0577 0609 0513 0513 0545 0513 0512 0545
0512 0513 0609 0545 0513 0608 0545 0512 0577
0545 0512 0577 0576 0545 0609 0577 0513 0512
0545 0512 0577 0608 0609 0512 0513 0545 0545
0577 0513 0545 0608 0577 0512 0609 0545 0513
0577 0545 0512 0513 0545 0545 0513 0512 0545

Looks like it defines them as 16x16?

Code Select Expand

0772 0773
0804 0805

[Daddy Poi's Oily Gorillas]

I probably should have put the magic numbers in Constants... But you know, it is a WIP editor, anyway.... so anything can happen.

Let's see if I can summarize a few things...
Form_Load will check if the last file you loaded exists, and will load it if it does. If not, a Open File Dialogue will pop up.
The ROM is loaded to an array called "rom" (I used ReadAllBytes to do that. That is in the Bits.openFile function I made- which may or may not be an unnecessary encapsulation, depending on what else is added to that function.)
OpenROM function is pretty much just version checking, but it will also call the loadEntireMap() function that we are interested in. (Same function also called when pressing Pg Up/Down...)

@loadEntireMap():
This beginning part, is coming straight from Atrius's editor. :P (To help with automatically supporting all ROM versions, but I did not do this for everything... like the text stuff.)
maps_pos1 is that primary table, and maps_pos2 is the one with the MFT indexes for map/pal/tilesets... (I call the World Map functions to keep those ones from being used in those cases/may need organizing.)

paldat = new byte[0x1C0]; //0x200];
            decompress(rom, pos[1], paldat, 0); initPal(paldat, 0, 0xE0); //pictureBox1.Image = PixelsToImage(pal, 16, 14, 16);
            tsetdat = new byte[0x10000];
            decompress(rom, pos[2], tsetdat, 0x0000);
            decompress(rom, pos[3], tsetdat, 0x4000);
            decompress(rom, pos[4], tsetdat, 0x8000);
            decompress(rom, pos[5], tsetdat, 0xC000);

This is the part that is decompressing the Palette and Tilesets used... initPal is for converting the values from 555bpp format GBA uses, to a 888bpp. (This is done because Windows 555bpp colors one isn't the same.... Although, with the blendit (blending function) I added once, I could probably change when this conversion happens, but it probably doesn't matter.)

stiles = new byte[0x10000]; tmap = new byte[0x10000]; visfx = new byte[0x200];
            //int length =
            decompTileData(rom, pos[0] + (Bits.getInt32(rom, pos[0] + 0x24)), stiles, 0); //8x8 tile table - Decompress & Deobfuscate.
            //Height Tile Data - Skip for now.
            decompMap2(rom, pos[0] + (Bits.getInt32(rom, pos[0] + 0x2C)), tmap, 0); //16x16 tilemap - Decompress & Deobfuscate.

This part will decompress the the map data and 8x8 mini-tiles...

Keep note of those arrays where the decompressed data is placed: paldat, tsetdat, stiles, and tmap.

The end of this function is basically error avoidance, so things may need cleaning up. And of course, I also call dispMap, which is supposed to take all that data, and make an array of pixels. PixelsToImage being set to a picture box should make that self-explanatory.  (Either that, or I have the function return a Bitmap... For these types of things, I like to have disp in the function's name.. Heheh.)

int tile = Bits.getInt16(stiles, t); t += 2;

                                        int q = (tile >> 4) & 0xC0;
                                        int tilepos = ((r * scnw) + c);
                                        //int k = q + 0x40;
                                        //int t2 = ((tile & 0x3FF) + ((rom[pos[0] + 7 + l] - 1) * 0x200)) << 5; //+ tset
                                        int t2 = ((tile & 0x3FF) << 5) + ((rom[pos[0] + 7 + l] - 1) * 0x4000); //+ tset
                                        if (t2 >= 0) {
                                            int tilePal = (tile >> 8) & 0xF0;

In other words, the tile flipping, 4-bit palette thing, 10-bit tile index thing, should be checked in the disp function.

int m = 0; int l = 3;
                while (m < 4) { //for (int m = 1; m <= 3; m++) {
                    l--; //int l = 2;
                    while ((l >= 0) && (rom[pos[0] + 4 + l] != m)) {
                        l--;
                    }
                    if (l < 0) { l = 3; m++; continue; }
                    int layiOffset = 1 << (3 - l);
                    int shiftx = rom[pos[0] + 0xC + (l * 8)], shifty = rom[pos[0] + 0xD + (l * 8)];
                    int shift = (shiftx & 0xfe) + ((shifty & 0xfe) << 7);
                    for (int row = 0; row < (scnh); row += 16) {
                        for (int col = 0; col < (scnw); col += 16) {
                            //int t = Bits.getInt16(tmap, 0x8000 + dx + shift + (dy << 7)) & 0xFFF;
                            int t = (Bits.getInt16(tmap, 0x8000 + (col >> 3) + shift + (row << 4)) & 0xFFF) * 8;

I think this is the part where you can see the separation between map layers...

[Triyence]

Has anyone created a GitHub repo for this?

[Daddy Poi's Oily Gorillas]

I doubt it?
And I'm not even sure if/how I can/could get use to using Github... hmm... It's nice to have, but the question is whether it is a nuisance to deal with or not.(?)

Edit: Perhaps someone has some suggestions on how to make it super simple? - Since I might have a look into it again at some point...

[Triyence]

I have a fair amount of knowledge of git. It definitely is something needed for true open source dev. Makes things like this much easier.

I made it so the layer buttons work, and I also added a button that copies the tile data that I needed to the clipboard. I have no idea who is tracking versions but if you want this here you go.

Quick screenshot:



Clicking the 'Copy Tile Info To Clipboard' button would give you this:

Code Select Expand

╔═════════╦════════╦═══════════╦═══════╦═══════╦═══════╦═══════╗
║ LayerId ║ TileId ║ PaletteId ║ TileX ║ TileY ║ VFlip ║ HFlip ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 772    ║ 0         ║ 0     ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 773    ║ 0         ║ 8     ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 804    ║ 0         ║ 0     ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 805    ║ 0         ║ 8     ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 774    ║ 0         ║ 16    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 775    ║ 0         ║ 24    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 806    ║ 0         ║ 16    ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 807    ║ 0         ║ 24    ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 772    ║ 0         ║ 32    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 773    ║ 0         ║ 40    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 804    ║ 0         ║ 32    ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 805    ║ 0         ║ 40    ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 774    ║ 0         ║ 48    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 775    ║ 0         ║ 56    ║ 0     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ 3       ║ 806    ║ 0         ║ 48    ║ 8     ║ FALSE ║ FALSE ║
╠═════════╬════════╬═══════════╬═══════╬═══════╬═══════╬═══════╣
║ ...       ║ ...      ║ ...         ║ ...     ║ ...     ║ ...     ║ ...     ║
╚═════════╩════════╩═══════════╩═══════╩═══════╩═══════╩═══════╝


[Daddy Poi's Oily Gorillas]

Well, I may have limited knowledge, so I am not sure of best practices... To me, it feels like sending files over Skype is one of the fastest/simple ways to do it. (But that's only in thought of the inexperienced mind.)

I do have a github account with a Yoshi Miagic repo uploaded to it... (I did not keep updating the repo for future edits, though.) https://github.com/charleysdrpepper/ ; But I think I might have done it with a program? (Unless I forgot.) So that is proof that I can at least get /something/ to work....

@Layer buttons: Okay, I may take a look at it... I do know that I did code some layer toggling in Yoshi Magic...(A newer version if not the one in the repo.) Where the same buttons also say which layer you are currently editing. (First you select the layer button to make it active, then you are suppose to click it again to toggle the layer's visibility.) Additionally, I also made it so you can use the number keys instead. 1,2,3...

and I also added a button that copies the tile data that I needed to the clipboard.

That works too, but I want to say that there came a point that when I check data in an array, I might save a file to the hard drive of its contents (using WriteAllBytes), and look at it from HxD. That's also easy to do with multiple arrays at a time.


---
@sel: I might have gotten that from Atrius's editor, probably? But I think it's ridiculously long to rename it to InitialMapSelectionIndex. ; Why not selectedMap or currentMap. (Or similar.) (I'd probably go for even shorter, though... for my preference... selMap/curMap, totalMaps are ideas. Since it is still very clear what it means.) However, I do understand that there is a are a number of coding styles going on around the net, but whether to follow them or not... can also be controversial, so I go with my own preference./Others go with theirs.

[Triyence]

How do I get the char base for a given layer? What part of the code is doing that?

Essentially this:



Which must be telling me where to index into the tileset to pull the tiles.

[Daddy Poi's Oily Gorillas]

int t2 = ((tile & 0x3FF) << 5) + ((rom[pos[0] + 7 + l] - 1) * 0x4000);

^ Within the map file header thing (That has the seven compressed files.)
Basically, char base is always divisible by 0x4000. (You can see the IO Viewer at around 0x04000008 for BG0 and onward... to notice it is 2-bit.)

[Triyence]

Thanks, that was indeed it. I used this to get an offset from the TileId that gets read:

Code Select Expand

(rom[pos[0] + 7 + l]) * 0x200

Values are either, 0, 512, or 1024.



Hopefully the last thing. Any idea why I'd be getting tiles like this around the edges?







The tiles are technically in no-mans land (not possible to view in game). However in gsmagic they get mapped to black tiles. I'm wondering where the discrepancy is coming from.

Edit: Oh yeah, another thing. How are transparent pixels dealt with? Seems like each tile would have a pixel color defined as transparent or something like that. I'm looking at where the code has blendflag and blendit usage, wondering if it's in there?

Looks like it's always the first color in the palette gets treated as transparent. Please correct me if I'm wrong.

[Daddy Poi's Oily Gorillas]

Well, the editor is by no means perfect...Considering the bugs I were to lazy to fix as of yet. :P Not only that,  but I didn't even indicate/make obvious one of the features it has: If you push 1, 2, or 3, on your keyboard... You can get another window up that displays the 8x8 tiles... If you push space, you'll see the used 16x16 tiles.... That was on my path to make this an actual Map Editor. - Which I have not done yet. Heheh. (It wouldn't be that difficult if I wasn't procrastinating... )

In-game, animated data is applied, in the editor, the animated tileset is decompressed, but it is not applied onto the tilesets... so it's possible that could be one scenario where tiles may go wrong... not saying that this applies to those particular tiles or not, though...  (But do note that the game can also have the tiles appear wrong anyway... if it is in an area not seen in gameplay.)


@blending: Basically, there are flags for which layers get applied to others... 1st target and 2nd target... (bldmod variable in the editor... you can also check GBA's I/O Viewer at around 0x04000050) ... and well.... a lot of the info is on GBATEK, but I guess since you striked it out, I don't have to go in-depth. - I can say that blendflag array thing... was my own solution.... (While trying to think through GBATEK's description of it.)

And yeah..,
-The first color in a palette that a tile can use is the transparent one. (So 0 = transparent.) Should be true for both 16 and 256 color palette selections....
-The backdrop is always the first color in the 256 color BG palette.




While I think readability is more important than speed (As long as it isn't too slow, that is)... I also think that organizing a project can be a bit difficult, so I generally prefer to hold that off until I have a better idea of all the code needed, so I can think out the refactoring strategies... (I'm also tempted to blame the coding language for it's messy formatting. And I want to compare it to Python to support that argument. :P)

[Triyence]

Thanks, that makes sense.

Is there a flag that tells me if a tile is animated? I.e. a flag that indicates that this tile is a water tile and thus, animated?

[Daddy Poi's Oily Gorillas]

A flag? Ehh... I think it's more "index to tile" probably...
There should be some animation data in there the fifth file of those seven compressed files...

02008000 = Map Code
02010000 = Tilemap (Including graphics, event indexes, heightmap, etc.)(32-bit a tile)
02020000 = Tile data (Mini-tiles) (F000=Palette; 0C00=Flips; 03FF=Tile)
02024000 = Heightmap tilemap.
02028000 = Animation Tileset
0202C000 = Heightmap tile table (Defines the 16x16 tiles that will be used in the heightmap)
0202D000 = Tile animation data (Stuff like water)
0202DE00 = Tile layer visual effects (Sun rays in Madra)
0202E000 = Coord Data
02030000 = Misc data. No files... but can be used anywhere in code, and sometimes linked from 03000000 list.
02040000 = Just to make it clear that this is where WRAM ends, and it's just mirrors from here.... (And the next RAM section is IRAM, which is very short. 0x03000000-0x03007FFF.)

[+0] = ?
[+2] = Index of segment in tilesets to get from. (0x0600-0x07FF = Animation tileset.)
[+4] = Number of tiles in segment.
[+6] = Index to put segment at.

[superjeenius]

It's not quite ready for public consumption yet, but in the spirit of open source I took some of my afternoon and implemented the Item Viewer in the C# game editor.  No support for editing, just for viewing right now.  Will upload the source when I iron out a few more bugs and finish adding in the text for use items and unleash names.

(I also had to implement the lion's share of the init_globals script too so most of the global variables will be intact.)

[Daddy Poi's Oily Gorillas]

Interesting to actually see someone actually working on this! Cool.
It's pretty simple for the most part, yeah? I usually end up procrastinating for some resaon, though... Maybe it is because of the mess. (Abstract or not.)

-Why parenthesis around item names? I think my plan would be to do it similar to the text editor. (Virtual list)... but where it'd only grab the item names list. It would also be formatted in the same way, most likely.

[superjeenius]

Interesting to actually see someone actually working on this! Cool.
It's pretty simple for the most part, yeah? I usually end up procrastinating for some resaon, though... Maybe it is because of the mess. (Abstract or not.)

-Why parenthesis around item names? I think my plan would be to do it similar to the text editor. (Virtual list)... but where it'd only grab the item names list. It would also be formatted in the same way, most likely.

Just temporary.  Just needed to get them in the listBox, no other reason than that.  I also figure that GUIs are easy to modify once the underlying structure is there, so I'm not terribly fussed about how it looks at the moment because that can easily be changed whenever, as long as it's all functional.

Having a small issue with importing the GSdatahandler dll, even with all the right arguments and the correct calling convention it is still acting up outside of debug (inside of debug it's fine).  We'll see what's up with that going forward.

[Daddy Poi's Oily Gorillas]

Oh yes, almost forgot to mention I think the text editor only works for GS2 at the moment... But it should easy enough to change GS2's pointers to GS1's to make it work for GS1.

--
@GSdatahandler.dll: Not needed for this C# project... But it was there for the GM one because of how bad GM is for some reason. (Speed issues and what not, if I recall.) - Atrius's editor is fully open-sourced (last version that was posted in News & Announcements forum.), and if you look at the code... It was mostly compression and/or decompression... and space manager stuff... etc.

[superjeenius]

@GSdatahandler.dll: Not needed for this C# project... But it was there for the GM one because of how bad GM is for some reason. (Speed issues and what not, if I recall.) - Atrius's editor is fully open-sourced (last version that was posted in News & Announcements forum.), and if you look at the code... It was mostly compression and/or decompression... and space manager stuff... etc.

It's so much simpler to use the code that's already been made as opposed to recoding the compression/decompression stuff for C# though because it all works fine...  I have the source for the GameMaker editor (I loathe GameMaker and their stupid scripting language...), but I'm very much a fan of going the 'path of least resistance'.  Down the line it might become easier to bring over all the code, but for now... meh.  That's just me though.

[Daddy Poi's Oily Gorillas]

I agree about the simpleness of ease it gives....

However, I do have a comp.cs with decompression/compression routines in them. (Thge text compression did not come from Atrius's editor though, it was my own code... that rewrites the char tables for best compression.)

Main point is that gsdatahandler is 35 KB... so it is very small... and if I left anything out, then how hard would the transferal be? Hmm... That, along with whether you need any of those functions anytime soon or not. At least when assuming C++ and C# are pretty similar (but not the same)... (Automatic conversion tool?)  And who knows if I'll consider converting this C# project into a C++ project one day... I kinda thought C# was a little simpler, though... but they are both good processor-fast languages. - Either way, it is imperative that it becomes multi-platform. (Keep putting off looking into Mono.)

Space manager stuff... I left out... I was thinking about having a somewhat different system.... For example... Atrius will repoint individual text sections, while I try to keep everything in order. (Not 100%ly the same order as the game, but that's probably because it'd take a little extra code that I probably didn't feel like putting in at the time. The text compression seems 1:1, though.)