IncBin Errors

Started by jsp, May 01, 2019, 20:15:16

Previous topic - Next topic

jsp

When trying to compile a project in debug mode where the main (file 1) and also an imported (file 2) uses IncBin it comes up sometimes (often) with errors:
multiple definition of `_bb_main_ib_3'
...  more errors if there are more IncBin lines in file 1

undefined reference to `_bb_test_file2_ib_1'
...  more errors for the IncBin lines of file 2

collect2.exe: error: ld returned 1 exit status
Build Error: Failed to link...

In release mode no errors and all imported media is available.

Is this a bug?
Strange that it does not always error. But it happens in 32 and 64 bit mode, threaded or not, GUI or console.
Deleting the .bmx makes no difference
Maybe my environment but rebuild all modules did not help.

Any idea?

Smallest way to reproduce on my system:
File 1:
Import "File2.bmx"
SuperStrict
Incbin "IncBinData/XSL0.txt"
'or more IncBin
Print "Main"


File 2:
SuperStrict
Incbin "IncBinData/ArrowDown.png"
Incbin "IncBinData/ArrowLeft.png"
Incbin "IncBinData/LangListBox.bmp"
Print "Imported"

-jsp-

Derron

while your error message sounds like an bug ... I at least am not able to reproduce it here. Compiled ~30 times (recompiled, non quick) and it worked and worked ...

Tried with "incbindata"-folder and without, added 3 incbins to each file...


./bmk makeapp -g x64 -t console -a -d -x "Testcodes/NG/incbin/file1.bmx"
[ 23%] Processing:file2.bmx
[ 23%] Processing:file1.bmx
[ 23%] Compiling:file1.bmx.debug.linux.x64.incbin.c
[ 24%] Compiling:file2.bmx.debug.linux.x64.incbin.c
[ 84%] Compiling:file2.bmx.debug.linux.x64.c
[ 85%] Compiling:file1.bmx.console.debug.linux.x64.c
[100%] Linking:file1
Executing:file1
Imported
main


Tried release and debug.
Are your bcc/bmk/brl/pub/... uptodate or are you using one of the releases?


PS: you are using a .bmp file - for a reason? (asked as png and jpg should be the better alternatives - or tga)


bye
Ron

jsp

Thanks for trying.

Unfortunately I did not find a better way to reproduce it absolutely safe.
If it compiles I can re-compile also without problem.
When I change something (in the code but not the IncBin) it re-compiles and the problem occurs it seems it does not go away until again the code changes.

Are your bcc/bmk/brl/pub/... uptodate or are you using one of the releases?
I use release 0.99.3.31 and the latest pub.mod sync via GIT.
But to be sure I did nothing wrong to pub.mod myself I just downloaded again the release version in a fresh folder, compiled file 2 and then file 1 and voila it comes up again with the same error.
Not sure what happens here.

QuotePS: you are using a .bmp file - for a reason? (asked as png and jpg should be the better alternatives - or tga)
No special reason, was just available, but you are right I should probably change it to png as well.
-jsp-

Yellownakji

Quote from: jsp on May 02, 2019, 00:24:05
Thanks for trying.

Unfortunately I did not find a better way to reproduce it absolutely safe.
If it compiles I can re-compile also without problem.
When I change something (in the code but not the IncBin) it re-compiles and the problem occurs it seems it does not go away until again the code changes.

Are your bcc/bmk/brl/pub/... uptodate or are you using one of the releases?
I use release 0.99.3.31 and the latest pub.mod sync via GIT.
But to be sure I did nothing wrong to pub.mod myself I just downloaded again the release version in a fresh folder, compiled file 2 and then file 1 and voila it comes up again with the same error.
Not sure what happens here.

QuotePS: you are using a .bmp file - for a reason? (asked as png and jpg should be the better alternatives - or tga)
No special reason, was just available, but you are right I should probably change it to png as well.


Did you build your modules as debug?   You should always debug-build modules, even if you don't use this mode.   Also, any reason you need to INCBIN?  Consider using an external binary solution.

Derron

Do you compile files which import file2.bmx or these incbins too?
Or hit compile on file2.bmx accidentally?

Do you build with quick/quick scan?

In such cases I have this duplicate definition errors in other code situations too .


Bye
Ron

jsp

QuoteDo you compile files which import file2.bmx or these incbins too?
Not sure if I understand that correct.
In the example file 1 imports file 2 and file1 is compiled as well.
File 2 is imported only once.
No other file IncBins the same media.

QuoteOr hit compile on file2.bmx accidentally?
No, not really. Actually I do have 70 imports or so in the current project. Every import is normally compiled, tested and run on it's own until I decide it can go into the main. Compiling those imports beforehand is quite normal for me. Most other files are imported more than once.
During development it can happen that an already integrated file needs adjustments and then it will be compiled again on its own before going further.

Nevertheless I can imagine that this is might be the trigger. I will monitor it more closely.

QuoteDo you build with quick/quick scan?
That option was my first guess, but it happens with and without. (and deleting the .bmx should fix it or not?)
Depending on what I change I switch it on, because the compile takes quite some time for all files.
As I am most of the time in release mode and seldom using debug it did not hurt, no other side effects so far.

QuoteDid you build your modules as debug?
Debug and release.

QuoteAlso, any reason you need to INCBIN?
Sure, the final app is supposed to be as small as possible and a single executable.
Actually the size could be smaller in NG.

-jsp-

Derron

Quote from: jsp on May 02, 2019, 12:11:24
QuoteDo you compile files which import file2.bmx or these incbins too?
Not sure if I understand that correct.

You answer that shortly later:


Quote from: jsp on May 02, 2019, 12:11:24
Every import is normally compiled, tested and run on it's own until I decide it can go into the main.

That can create issues with the way NG currently works. If you do that you might need to "rebuild" your main file everytime ( so that it recompiles all non-modules it imports too).
Especially if you compile files which import files on their own - and are imported then by "main.bmx" you will run into issues.

Did not experience this for incbin'd files (am not using this feature) but somehow NG decides to put some definitions in certain files (eg the "main"-file of the current compilation process). Now you compile "main.bmx" and ... NG sees that it does not need to recompile "importedfile.bmx" as it was compiled before. But the precompilates now contain information which _else_ would be placed into the precompilate of "main.bmx". It compiles "main.bmx" and tries to put things together ... bam, duplicate definition of something.


Quote from: jsp on May 02, 2019, 12:11:24
QuoteDo you build with quick/quick scan?
That option was my first guess, but it happens with and without. (and deleting the .bmx should fix it or not?)
You would need to ensure removing all the .bmx folders:
/main.bmx -> /.bmx
/sources/importfile.bmx -> /sources/.bmx

But else: yes, removing the precompilates should solve that issue if it was the same/similar problem to what I described above.


Quote from: jsp on May 02, 2019, 12:11:24
Actually the size could be smaller in NG.
- already using "Framework" command?
- on linux/mac (and Windows surely too) you could use "strip" to remove unnecessary stuff from the executable
- use 7zip to compress your release (packs much tighter than zip)

- replace some modules (eg using the new "brl.xml" [tiny xml] instead of "bah.libxml" removes some hundred kilobytes)


bye
Ron

jsp

Thanks, I will keep your explanation in mind. Very useful.

The size is not yet a problem, but the project will for sure increase. Nevertheless the smaller the better as the file will be transferred sometimes over very bad connections.
I used UPX before to get the size down but need to move away now as it removes the command-line interface and that is something I need in the near future.
Can't remember how good 7zip packed. Need to download the latest version and check.

Quoteyou could use "strip" to remove unnecessary stuff from the executable
Do you have an example? I can't remember that strip was part of BlitzMax as Framework was. Is this a new feature in NG?


-jsp-

Derron

#8
On Linux/Mac you just access it via commandline.

Eg. I have this file "post.bmk" in my project directory, BMK will execute it after compilation:

@define doPostInstall
# LINUX/MAC: strip binary when doing a releasebuild
if bmk.Platform() == "macos" or bmk.Platform() == "linux" then
if bmk.IsDebugBuild() == 0 then
if bmk.Platform() == "linux" then
bmk.Sys( "strip --strip-all " .. %exepath% .. "/" .. %outfile% )
elseif bmk.Platform() == "macos" then
bmk.Sys( "strip " .. %exepath% .. "/" .. %outfile% )
end
end
end
@end

# run the post install
doPostInstall

(only for release as it else would strip the debug information).

It is easy to add a "windows" variant - but it depends on your MinGW path - eg.:
MinGW32x86\bin\strip.exe


Edit:

@define doPostInstall
# WINDOWS/LINUX/MAC: strip binary when doing a releasebuild
if bmk.IsDebugBuild() == 0 then
if bmk.Platform() == "linux" then
bmk.Sys( "strip --strip-all " .. %exepath% .. "/" .. %outfile% )
elseif bmk.Platform() == "macos" then
bmk.Sys( "strip " .. %exepath% .. "/" .. %outfile% )
elseif bmk.Platform() == "win32" then
bmk.Sys( bmk.MinGWBinPath() .. "\\strip.exe --strip-all " .. %exepath% .. "\\" .. %outfile% )
end
end
@end

(untested - maybe you need to use "/" instead of the encoded backslash "\\".
If NG is already instructed to remove that stuff then you won't see much decrease of the files after stripping.

bye
Ron

Yellownakji

Quote from: jsp on May 02, 2019, 12:11:24
..

Using INCBIN won't make the EXE smaller.   Sure, a single executable is nice, but you'll essentially double your ram, as INCBIN loads files into ram and then decompresses them when needed. so...

jsp

Thanks Ron, very nice.

QuoteSure, a single executable is nice, but you'll essentially double your ram...
I know, but a single executable is more important for me at this time.
Maybe it changes in the future at some point who knows.
As most people create games here I must say this project is mainly for Windows server with 32GB of RAM.
So the bad internet connection the file has to travel through is a bigger bottleneck than the RAM I use. Actually the app may take 1GB when running so no problem with a very few kb from the IncBin.


At the end I did not test all stuff yet, but looks good so far and I am quite happy with NG.
My main stuff uses a lot of external processing and background threads and my LogicGui ListView (created an NG compatible version and changed pub.mod for the additional methods I needed) and it all turned out ok. The GC cleans up a bit different.
Very good job done here (Thank you to Brucey and all involved) and I decided now to transfer my project from BlitzMax to NG.
So some more work until fully converted...

-jsp-

Derron

While you convert you might think of utilizing the new features for more performance / "better" code:
- structs
- interfaces
- method/operator overloading
- some new modules to provide more efficient string processing (TLowerString for comparison and StringBuilder.mod for appending to strings efficiently)
- ...

I saw your changes for maxgui - dunno what you needed to change in pub.mod ?

bye
Ron

jsp

QuoteWhile you convert you might think of utilizing the new features for more performance / "better" code:
Yep, sounds great.
Do we have some docs how those stuff is implemented in NG?

Quotedunno what you needed to change in pub.mod
I added stuff for LVCOUMN and LVITEM in commctrl.cpp and -bmx. And some more to come I think. Just did the main stuff to see if the project is working.
This is not used in plain MaxGUI and was probably thus not implemented yet.
When ready I can create a pull request to get into the main branch, or if you like already now what I have so far.
-jsp-

Derron

https://blitzmax.org
There is a docs section. If it misses something you like to know... File an issue at github.com/bmx-ng ...


@ like
I only scanned through and think some of this stuff needs or at least could benefit of a bit refactoring. Also I am not sure about certain widgets ... Some might have native implementations on Mac/Linux ... Some not.

Bye
Ron