News:

The forum has been updated to SMF (2.1.3)!
Please be patient as we work to polish up the place and update features as we can.

Main Menu

Making text editor.

Started by Lord Wolfram, 25, June, 2014, 01:34:18 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

Lord Wolfram

Alright team we right now are waiting for Leaf to Fix our script.
While messing around with fire emblem editor I finally understand what Pablitox meant.
We must find scripts take them out then replace with translated ones and here you go our project is done.
sounds simple but it is not because as we know that we must double space.Make good tools.
I was about to start make ones in java. and use them.
Oh one thing is there a way to increase free space using hex?
and of curse we have new member of translation project: Batsquid.

pablitox

#81
Quote from: Lord Wolfram on 31, July, 2014, 02:58:33 PM
Alright team we right now are waiting for Leaf to Fix our script.
While messing around with fire emblem editor I finally understand what Pablitox meant.
We must find scripts take them out then replace with translated ones and here you go our project is done.
sounds simple but it is not because as we know that we must double space.Make good tools.
I was about to start make ones in java. and use them.
Oh one thing is there a way to increase free space using hex?
and of curse we have new member of translation project: Batsquid.


a new translation member? awesome!. though I'm inclined to ask if they have good knowledge of the summon night swordcraft story series.                                                                  

It relies on various things from older games so the bettet the knowledge couls be awesome.

as far as things goes i think that aent taking care of the hacking part as he must be decyphering the missing opcodes. that is an useful part. as you say now we need to find a way to make the script fit.

Aent

#82
Quote from: pablitox on 31, July, 2014, 04:06:59 PM
as far as things goes i think that aent taking care of the hacking part as he must be decyphering the missing opcodes. that is an useful part. as you say now we need to find a way to make the script fit.

I've been working on it on and off in my spare time, but I am pretty new to all this. That tutorial you posted was quite helpful so my thanks to whomever wrote it and to you for posting it.

Ideally I want to break the limit of the current text blocks(PSI3 segments which is what is shown in my last post) so we can have a fuller translation. As it is now, each text block links to the others either by an index number to an array of pointers or a pointer of it's hex location. Still investigating that. If for instance, you take one segment, say the mine entrance right after the intro, and replace it with another. As soon as you leave after fighting VE it will warp you to the new location instead of the mine. In my case it was from there to the shop in the first town. If I walked out of the shop then, it took me outside into the town as it normally would. This tells me that the locations that they lead to aren't linked to the current text blocks location in the game file. If we were to pull out all the text blocks, put our full translation in and put them back in, they would run out of room. Sort of like putting 4 rectangles where 4 squares are supposed to fit. The way to fix this is put them in as normal and once we run out of room for the next one to fit, move it to free space and adjust the pointers as needed for all the text blocks. With luck it will be some sort of array, but I'm not sure honestly. I have located a value that if changed links to various locations and battles but more investigating is needed. As for the max rom space issue of 32mb, I'm not sure how close we'll come to that. I forget how much free space was said in the other thread. Something like 280kb? I'll have more on that situation soon I hope.

Download here.
I'm gonna include the program I use to dump the text blocks as well as a .bat file to use with DSDecmp. Just unzip to a folder with "sn3.gba" in it.(without the quotes). Doubleclick dumptexts.exe and it will dump the encrypted texts to the folder "output". Their names will be in the form of "decimal_hex". Then, with DSDecmp.exe in the original folder, doubleclick dsdecmp_folder.bat and it should make a folder called output_dec. Those will be the decompressed ones as shown in my last post. They include the original script, events, etc, albeit in a fairly unreadable form except for the actual script. For those of you who wish to see the code of dumptexts.exe, it's in src. You'll need GCC to compile most likely. I plan to include any programs/tools I create and their source as I don't want my progress for better or worse to be lost if something beyond my control should happen, lol. Beware I have some experience with C, but am certainly no expert and use google quite often.

If any questions/comments/tips/criticisms, etc I'll try and be more active here to see them. I can't promise I'm the answer to what you've been looking for, but I'm willing to try and help, lol. My apologies for the late response as well as the text-wall.


p.s. Offsets.txt will be created as well. It's just a list of all the starting locations of the compressed segments in hex. Also, not all the txt files will be story dialogue. Some are just empty rooms or lead to battles and such. Also, some might be two different parts of one location. For instance, in the first game I had one for the intro dialogue and one for map items. ("...may..." cutscene for the dialogue and in a separate text for picking up the entrance ticket as well as the placement and teleport location of the stairs)

pablitox

Quote from: Aent on 01, August, 2014, 01:04:56 AM
Quote from: pablitox on 31, July, 2014, 04:06:59 PM
as far as things goes i think that aent taking care of the hacking part as he must be decyphering the missing opcodes. that is an useful part. as you say now we need to find a way to make the script fit.

I've been working on it on and off in my spare time, but I am pretty new to all this. That tutorial you posted was quite helpful so my thanks to whomever wrote it and to you for posting it.

Ideally I want to break the limit of the current text blocks(PSI3 segments which is what is shown in my last post) so we can have a fuller translation. As it is now, each text block links to the others either by an index number to an array of pointers or a pointer of it's hex location. Still investigating that. If for instance, you take one segment, say the mine entrance right after the intro, and replace it with another. As soon as you leave after fighting VE it will warp you to the new location instead of the mine. In my case it was from there to the shop in the first town. If I walked out of the shop then, it took me outside into the town as it normally would. This tells me that the locations that they lead to aren't linked to the current text blocks location in the game file. If we were to pull out all the text blocks, put our full translation in and put them back in, they would run out of room. Sort of like putting 4 rectangles where 4 squares are supposed to fit. The way to fix this is put them in as normal and once we run out of room for the next one to fit, move it to free space and adjust the pointers as needed for all the text blocks. With luck it will be some sort of array, but I'm not sure honestly. I have located a value that if changed links to various locations and battles but more investigating is needed. As for the max rom space issue of 32mb, I'm not sure how close we'll come to that. I forget how much free space was said in the other thread. Something like 280kb? I'll have more on that situation soon I hope.

Download here.
I'm gonna include the program I use to dump the text blocks as well as a .bat file to use with DSDecmp. Just unzip to a folder with "sn3.gba" in it.(without the quotes). Doubleclick dumptexts.exe and it will dump the encrypted texts to the folder "output". Their names will be in the form of "decimal_hex". Then, with DSDecmp.exe in the original folder, doubleclick dsdecmp_folder.bat and it should make a folder called output_dec. Those will be the decompressed ones as shown in my last post. They include the original script, events, etc, albeit in a fairly unreadable form except for the actual script. For those of you who wish to see the code of dumptexts.exe, it's in src. You'll need GCC to compile most likely. I plan to include any programs/tools I create and their source as I don't want my progress for better or worse to be lost if something beyond my control should happen, lol. Beware I have some experience with C, but am certainly no expert and use google quite often.

If any questions/comments/tips/criticisms, etc I'll try and be more active here to see them. I can't promise I'm the answer to what you've been looking for, but I'm willing to try and help, lol. My apologies for the late response as well as the text-wall.


p.s. Offsets.txt will be created as well. It's just a list of all the starting locations of the compressed segments in hex. Also, not all the txt files will be story dialogue. Some are just empty rooms or lead to battles and such. Also, some might be two different parts of one location. For instance, in the first game I had one for the intro dialogue and one for map items. ("...may..." cutscene for the dialogue and in a separate text for picking up the entrance ticket as well as the placement and teleport location of the stairs)


That was certainly helpful! I can't thank you enough aent! this looks good.

What we should do with the translated script? it belongs to other dump, so i guess we should need to export it to yours and translate the missing bits of text, is that right?

Lord Wolfram

Hold a sec!
Leaf is doing some work with scripts
so we must wait for him to show up.
And then we put scripts little by little in forum so we can work easier on project.

pablitox

Quote from: Lord Wolfram on 01, August, 2014, 12:48:34 PM
Hold a sec!
Leaf is doing some work with scripts
so we must wait for him to show up.
And then we put scripts little by little in forum so we can work easier on project.


yup, you're right.

BlackDog61

#86
Hi!
So many things happening - good job, Aent!
I was under the impression that the PSI3 marker might have a length shortly after it. Doesn't it?
(If it did, then that would explain why there's often a lot of garbage after the dumped text. ;)

You're probably already planning everything, so just a humble suggestion: how about starting to encapsulate non-text stuff in XML-like tags? In the end, that's probably how you'll want to isolate opcodes from the rest of the text.

I don't think Aent needs to wait for anyone else's return to keep parsing the opcodes. Or did I miss anything?

EDIT: I assume you've already looked into the file called "Dialogue enconding.txt"? It tells about some opcodes that were parsed by previous project members. Here is a quote:
Here is a (likely incomplete) list, from my own observations:

    02 00 xx xx - conditional jump, for example to Rif's text.

    03 00 xx xx - jump, for example to merge back from story arc.

    07 00 xx xx - Function call, for example to set portaits, etc
                  Returns from function with 08.

    13 03 xx xx and
    14 03 xx xx - jump when the menu item is selected.


Hopefully that's useful. ;)

pablitox

Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
Hi!
So many things happening - good job, Aent!
I was under the impression that the PSI3 marker might have a length shortly after it. Doesn't it?
(If it did, then that would explain why there's often a lot of garbage after the dumped text. ;)

You're probably already planning everything, so just a humble suggestion: how about starting to encapsulate non-text stuff in XML-like tags? In the end, that's probably how you'll want to isolate opcodes from the rest of the text.

I don't think Aent needs to wait for anyone else's return to keep parsing the opcodes. Or did I miss anything?

EDIT: I assume you've already looked into the file called "Dialogue enconding.txt"? It tells about some opcodes that were parsed by previous project members. Here is a quote:
Here is a (likely incomplete) list, from my own observations:

    02 00 xx xx - conditional jump, for example to Rif's text.

    03 00 xx xx - jump, for example to merge back from story arc.

    07 00 xx xx - Function call, for example to set portaits, etc
                  Returns from function with 08.

    13 03 xx xx and
    14 03 xx xx - jump when the menu item is selected.


Hopefully that's useful. ;)

Oh yeah, this is the .txt

https://github.com/wangds/sncsm3/blob/master/resources/Dialogue%20Encoding.txt

Aent

#88
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
I was under the impression that the PSI3 marker might have a length shortly after it. Doesn't it?
Yep, the 2 bytes right after are the length of the uncompressed text. The bytes between the compressed PSI3 segments are all 0x00's and are there for padding to align to 0xC for the next PSI3 segments.
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
You're probably already planning everything, so just a humble suggestion: how about starting to encapsulate non-text stuff in XML-like tags? In the end, that's probably how you'll want to isolate opcodes from the rest of the text.
I actually planned on dumping only the script and it's offsets for the translation part, but got caught up in other work and haven't got back to it just yet. The PSI3 segment dumps were more just to show that some progress was being made and I was still working on it. =p I'll try and come up with something more suitable when we're a bit further along. The guys working on the text should keep up with however they're doing it and when it comes time to adapt it to a standardized way for it to insert the text, I'll do it.
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
EDIT: I assume you've already looked into the file called "Dialogue enconding.txt"? It tells about some opcodes that were parsed by previous project members.
Yep, and it's been a good help. There's still quite a few I need to check out mostly dealing with stuff that's not in the first two cutscenes, lol. In particular the various guardian beasts. I'll get back to them soon.


The main reason I wanted to post today is I solved one of the major hurdles we were facing with the locations of the pointers to the PSI3 segments. As I mentioned before, we were limited to the sizes of the compressed PSI3 segments because if they got even 1 byte bigger, they would overlap others causing a crash. My other post explains it all a bit better, but the good news is I'm able to move any PSI3 segment to a new location, including the end of the data(282kb free space roughly). So, when we put all the translated ones in, I'll be able to re-point to their new locations; whether it be adjusted down to accommodate newly added text or at the end of the file.

I'll write a small explanation on how it's done exactly sometime soon. I just need to test it a bit more.

Aent

#89
Doh, hit quote instead of modify. lol. Relevant post is the one before this.

Would a kind moderator please delete this post?  :sad:

pablitox

Quote from: Aent on 05, August, 2014, 07:55:00 PM
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
I was under the impression that the PSI3 marker might have a length shortly after it. Doesn't it?
Yep, the 2 bytes right after are the length of the uncompressed text. The bytes between the compressed PSI3 segments are all 0x00's and are there for padding to align to 0xC for the next PSI3 segments.
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
You're probably already planning everything, so just a humble suggestion: how about starting to encapsulate non-text stuff in XML-like tags? In the end, that's probably how you'll want to isolate opcodes from the rest of the text.
I actually planned on dumping only the script and it's offsets for the translation part, but got caught up in other work and haven't got back to it just yet. The PSI3 segment dumps were more just to show that some progress was being made and I was still working on it. =p I'll try and come up with something more suitable when we're a bit further along. The guys working on the text should keep up with however they're doing it and when it comes time to adapt it to a standardized way for it to insert the text, I'll do it.
Quote from: BlackDog61 on 01, August, 2014, 02:22:43 PM
EDIT: I assume you've already looked into the file called "Dialogue enconding.txt"? It tells about some opcodes that were parsed by previous project members.
Yep, and it's been a good help. There's still quite a few I need to check out mostly dealing with stuff that's not in the first two cutscenes, lol. In particular the various guardian beasts. I'll get back to them soon.


The main reason I wanted to post today is I solved one of the major hurdles we were facing with the locations of the pointers to the PSI3 segments. As I mentioned before, we were limited to the sizes of the compressed PSI3 segments because if they got even 1 byte bigger, they would overlap others causing a crash. My other post explains it all a bit better, but the good news is I'm able to move any PSI3 segment to a new location, including the end of the data(282kb free space roughly). So, when we put all the translated ones in, I'll be able to re-point to their new locations; whether it be adjusted down to accommodate newly added text or at the end of the file.

I'll write a small explanation on how it's done exactly sometime soon. I just need to test it a bit more.

That's a pretty good job Aent, please keep it up! Also, I saw that the english script is (almost) twice the length of the japanese one, will those 282 kb be enough for the script? Or we'll have to solve it any other way? I'm asking out of curiosity.

About the offsets, a spanish user by the name of CUE said that they could be of 7 bits, and they could be between 00 and 1F. What happens once we find them? should be test them to see where they lead to?

Aent

Quote from: pablitox on 05, August, 2014, 11:21:28 PM
That's a pretty good job Aent, please keep it up! Also, I saw that the english script is (almost) twice the length of the japanese one, will those 282 kb be enough for the script? Or we'll have to solve it any other way? I'm asking out of curiosity.
That's a bit tough to answer right now. With compression, adding in opcodes for linebreaks, etc, it can vary greatly. I'm going to switch it to reading 1 byte instead of 2 for each character. I plan on doing that as my next priority. I ran a few numbers and we should be alright, but I won't say anything for certain atm.
Quote from: pablitox on 05, August, 2014, 11:21:28 PM
About the offsets, a spanish user by the name of CUE said that they could be of 7 bits, and they could be between 00 and 1F. What happens once we find them? should be test them to see where they lead to?
Once all of them are found I'm going to redump the script with hex offsets to exactly where the opcodes are so if they need changed, I can change them as we go. For instance if we add 10 more characters to a line, it will count this and adjust all the locations of text and such accordingly. From there, I'll probably insert a bunch of random script just to test and make sure everything's working OK and that I didn't miss any opcodes. Then if everything seems alright, the proper script gets inserted next.

My current plans for myself:
1) Get 1 byte character reading working.
2) Finish opcode duty
3) Work on a proper dumper with hex offsets and manually edit in some script
3) Make an automated inserter using the hex offsets.
4) Insert random long script into game to test size and missing opcodes
5) Work on a variable width font? This would be a bit of a learning experience.
6) Adjust proper script to work with the inserter

There is of course more to it. I haven't looked into the graphics editing and "nameplates" yet. My main focus atm is just getting a working script into the game.

pablitox

That sounds like a solid plan of action ! I completely agre with it and yeah, the graphics part will be done after that.

leaf

Hello guys. I'm back from my trip now, so I can get back to work on editing. Unfortunately, I did not get a whole lot done while I was away, so I don't have anything new for you guys yet.

Can I ask what kind of timetable you guys are looking at? If you guys are trying to get this done quickly, I can speed up my editing by only going through the original japanese when the context is uncertain. What I've been doing is reading the original japanese, reading the provided translation, and then making edits as necessary. It's been a nice way to work on my japanese, but I'll admit it's horrendously slow compared to normal proofreading.
[spoiler=quotes]
[9:00:50 PM] Randel Peltier: Ok...what did I do last night?
[9:01:19 PM] Kain: Something boring, repetitive, and lasted for about sixty seconds.
---
[10:45:08 AM] Salanewt: But yeah, the elemental phalluses are being... Stroked up by Saturos and co., and the energy will go towards... Mt. Muffin, where the Golden Climax will arise.
[7:28:42 PM] Salanewt: An added bonus is that Isaac and co. were trying to stop Saturos and co. because their beliefs state that Mt. Muffin should remain a virgin.
---
[9:54:21 PM] Randel Peltier: Guess the number in my head an you get to pick what I say. Number between 1-10
[9:54:28 PM] leaf: 11
[9:54:36 PM] Randel Peltier: @#$%!
---
[8:38:13 PM] Randel Peltier: Shes like this queen up on a pedastal that I have yet to court.
[8:38:29 PM] Kain: You've tried courting her.
[8:38:43 PM] leaf: and failed spectacularly
[8:38:44 PM] Randel Peltier: Ive tried...shes the best dating game ever.
---
[12:24:35 AM] Salanewt: I need to find a picture of a naked person to put on the Christmas tree next year.
---
[2:19:06 PM] Zeadra: wait... Rief's a guy???
---
[1:09:57 PM] Zeadra: well if you want to know if its a new effect or something weird, just check GS1, if side step is there maby it is the nimble dodge thing
[1:10:35 PM] Kioll: For once, you've contributed something useful.  o.O[/spoiler]

pablitox

#94
Quote from: leaf on 07, August, 2014, 01:35:29 AM
Hello guys. I'm back from my trip now, so I can get back to work on editing. Unfortunately, I did not get a whole lot done while I was away, so I don't have anything new for you guys yet.

Can I ask what kind of timetable you guys are looking at? If you guys are trying to get this done quickly, I can speed up my editing by only going through the original japanese when the context is uncertain. What I've been doing is reading the original japanese, reading the provided translation, and then making edits as necessary. It's been a nice way to work on my japanese, but I'll admit it's horrendously slow compared to normal proofreading.

Yo leaf! Long time no see !

About the timetable, the thing is we dont have one, i will need to ask around to be sure, but the only thing we have is a google sheet with the common terms used in the translation.

Though you can use this site to get a good day by day reference of the game (up until 8 day, there're 12 if I remember correctly)
http://swordcraftfuntimes.blogspot.com.ar/?m=1

Edit: I don't know if it's a good idea or not but if multiple proofreaders will work on this, could it be a good idea to upload the files to github? that way all the proofreaders can see the changes.

I was thinking of changing this phrase 疲れが取れた! to You feel refreshed! as it's the one used in swordcraft story 2, I say it because there aresome lines were it's translated differently, meanwhile I'll locate the phrases and when it's done I'll write them here.


pablitox

Quote from: Lord Wolfram on 17, July, 2014, 12:46:35 AM
Alright things are ongoing that's good to hear.
And Fox can be right we must upgrade Ritchburn's tools.
Or maybe something like I trough Text editor.

I doubt that will be necessary, as ritchburn tools are kinda inneficient, or so I was told. Aent in on the job of getting opcoded and reducing character size, so I bet it will be a long task, as there must be a lot of opcodes between 00 and 1F. I believe whenthey have made good progress they will post here.

Lord Wolfram

You know how old this post is....
anyway when the scripts will be rewritten by leaf than If it will be necessarily we should change
[it is to it's] and so go on...

pablitox

Quote from: Lord Wolfram on 15, August, 2014, 12:13:57 PM
You know how old this post is....
anyway when the scripts will be rewritten by leaf than If it will be necessarily we should change
[it is to it's] and so go on...

I do? just wanted to reply your post.

Anyway yeah, leaf is already taking care of that, which is good. I played the second game to make use of system dialogues and put them on the third game.

Aent

Quote from: Aent on 06, August, 2014, 04:41:34 AM
My current plans for myself:
1) Get 1 byte character reading working.
2) Finish opcode duty
3) Work on a proper dumper with hex offsets and manually edit in some script
3) Make an automated inserter using the hex offsets.
4) Insert random long script into game to test size and missing opcodes
5) Work on a variable width font? This would be a bit of a learning experience.
6) Adjust proper script to work with the inserter

1) I've narrowed down this quite a bit. I'm literally learning a bit of ASM as I go, so has been quite a fun experience for me.
2) Did most of the important ones such as dialogue, female text, some jumps, etc.
3) I have a dumper that dumps the dialogue of the compressed texts. Think of something like we have already, except it has the offsets in it. This way I know exactly where to put the texts and the new locations once we make the strings longer.
4-6) Still working on.

For the most part, my biggest challenge is finding time throughout the week, lol. I've done most of this work on the weekend. Heopfully I'll have something a bit more flashy to show, like screenshots or something next time. =p

pablitox

#99
Quote from: Aent on 20, August, 2014, 02:17:41 PM
Quote from: Aent on 06, August, 2014, 04:41:34 AM
My current plans for myself:
1) Get 1 byte character reading working.
2) Finish opcode duty
3) Work on a proper dumper with hex offsets and manually edit in some script
3) Make an automated inserter using the hex offsets.
4) Insert random long script into game to test size and missing opcodes
5) Work on a variable width font? This would be a bit of a learning experience.
6) Adjust proper script to work with the inserter

1) I've narrowed down this quite a bit. I'm literally learning a bit of ASM as I go, so has been quite a fun experience for me.
2) Did most of the important ones such as dialogue, female text, some jumps, etc.
3) I have a dumper that dumps the dialogue of the compressed texts. Think of something like we have already, except it has the offsets in it. This way I know exactly where to put the texts and the new locations once we make the strings longer.
4-6) Still working on.

For the most part, my biggest challenge is finding time throughout the week, lol. I've done most of this work on the weekend. Heopfully I'll have something a bit more flashy to show, like screenshots or something next time. =p

That's a lot of good progress! Thanks Aent!

I do know about being busy too, as I started Uni again, so not gonna connected as much as I used to be. Anyway that's a lot of progress! Looking forward to some screenshots too!

Oh yeah, I wanted to ask you a question regarding system messages. I used Ritchburn tools to translate some missing system messages, like this:

In Main.c:

patch_str(0x000baf84,  16, "Buy how many?");
patch_str(0x000baf98,  16, "Sell how many?");
patch_str(0x000bafac,  18, "Create how many?");
patch_str(0x000bafbe,  16, "Use how many?");
patch_str(0x000bafd4,  22, "Is it okay to sell?");
patch_str(0x000bafea,  22, "Equip it right away?");


Here's a screenshot:





I hope it does not get in the way of the dialogue translation.My question is, is there a method to do this that does not involve trial and error? Usually when using the wrong hex address the string overlaps with another one, but besides that no other bugs.

Thanks for the good job!