[BMax 1.50] Figuring out memory usage

Started by Adam Novagen, March 17, 2021, 02:27:24

Previous topic - Next topic

Adam Novagen

Hey all,

Is it possible to get a reliable and accurate number for the total memory usage of a classic BMax program? SizeOf() seems borderline useless to me, since it counts basic ints and floats and the like, but not images or banks or audio... I could do an estimate the old-fashioned way using width * height * 4 to figure out the byte size of an image, and of course BankSize for banks, but that still leaves me unable to track memory usage of TSound audio files.

I thought I was on to something when I found BlitzSupport's reply at the bottom of this post that was archived from bb.com, but the PrintProc() function seems out of whack since it reports 0 for memory usage on any and all processes, and thinks Firefox is somehow using 57 threads... Any ideas?
We all know the main problem with dictionaries is that they contain too many words, and not enough butterscotch sauce!

Derron

#1
If it was only for the GCMemory you could always use GCMemAlloced().

Little side information:
I know it is not what you are looking for - but with NG you could open ub blitz.mod/blitz_gc.h and enable bbGCAllocCount.
this would allow you to bring this "extern" variable into bmx and to count what elements are created (and maintained) by the GC.
Also you could dump out what the GC was collecting - I used it to create a table exposing if I leaked objects (kept references accidentally)

https://github.com/TVTower/TVTower/blob/master/source/Dig/base.util.bmxng.objectcountmanager.bmx

---

On Linux you could execute some of the available commands to grep the amount of memory your processes (if running in threads etc) use. But seems you ask for Win32. Aren't there some Win32-api commands available for your desired information?


Edit:
https://stackoverflow.com/questions/63166/how-to-determine-cpu-and-memory-consumption-from-inside-a-process

There is a example on how to retrieve total virtual memory - and also how to retrieve virtual memory used by the process


As you only need this particular information you might just copy the example code in a "getmemory.c" file (the "logic" wrapped inside a function returning the value) which you include in your application (inside the "extern" part).



bye
Ron

Derron

#2
Dunno if I did it right (especially the BBLONG cast - as you talked about a "classic" program I think you might compile with legacy blitzmax - which has no "size_t" support).


Code (Blitzmax) Select

SuperStrict
Framework Brl.StandardIO
?Win32
Import "processmemory.win32.c"
Import "-lpsapi" 'add linker information so PSAPI lib is linked!
?

?Win32
Extern "C"
Function GetVirtualMemUsed:Long() = "GetVirtualMemUsed"
End Extern
?
?not win32
'TODO: add stuff for other OS
Function GetVirtualMemUsed:Long(); Return 0; End Function
?

For Local i:Int =  0 Until 10
Print "used: " + (GetVirtualMemUsed()/1024) +"kb"
Delay(10)
Next


And "processmemory.win32.c":
Code (C) Select

#include "windows.h"
#include "psapi.h"
#include "brl.mod/blitz.mod/blitz.h" //To access BBLONG

BBLONG GetVirtualMemUsed() {
PROCESS_MEMORY_COUNTERS_EX pmc;
GetProcessMemoryInfo(GetCurrentProcess(), (PROCESS_MEMORY_COUNTERS*)&pmc, sizeof(pmc));
size_t virtualMemUsed = pmc.PrivateUsage;

//vanilla blitzmax has no Size_T ...
return (BBLONG) virtualMemUsed;
}


If you really just run vanilla Blitzmax - size_t will be 32bit (as vanilla only outputs 32bit binaries). In that case you could switch from ":Long" to ":Int" and so return "BBINT" and not "BBLONG" .

Adam Novagen

This looks very useful, thanks Derron!

I believe I'll need some extra instruction on how to implement it though. I've used a good deal of BlitzMax's potential over the years but never really interacted with building modules, and frankly I'm not sure what to put where or how. I would assume I would copy your C code to a file named processmemory.win32.c and put it in uh... Somewhere in the \mod directory of the BlitzMax install? Do I have to compile the file at all, or just drop it in a new folder with the .BMX file and rebuild the modules? These are probably daft questions but like I said, never done anything with modules  :))

To answer your version question, yes, I am using to legacy BlitzMax here, simply because the game engine I'm working on is about six years deep at this point and I'm not confident in being able to port it to NG successfully; I could probably copy-and-paste but we're talking 43 separate files, in addition to Brucey's old GME module (the openAL version) for playing .VGM music and some old, finicky DX9 render-to-texture code from Kryzon. Obviously with legacy BMax being Win32, the program is limited to about 1.75GB before crashing with the old EXCEPTION_ACCESS_VIOLATION (ran a bunch of tests using banks, 'twas fun).

This won't be a problem for the game proper, it's a stylized RPG in the manner of major Mega Drive games of yore and I can probably keep the entirety of its content down to well under 1GB, but the engine is designed to be universal and allow for modding, custom content, etc. To make this process simpler, faster and more stable for the potential modder, the game indexes and loads all resources at startup, with the possible exception of audio. The downside of this is of course that the more content the engine has available to it, the more it will quickly start to eat memory, the main culprits being maps (8 bytes per tile, an average area would probably be about 64x48 [24KB] with areas as large as 1024x512 [4MB]) and images (at 32bpp, we're looking at 256KB per map tileset (always 256x256px) or 1MB for an average 512x512 sprite sheet.)

I do currently have a crude system in place that basically creates a blank file labeled "LOADING" while the game boots up, and then deletes it as soon as everything loads; if the game detects this file when it starts, it warns the player that it crashed during its last load attempt and that the crash was most likely due to excess memory usage. I would just like to have something a little better and more reliable in place so that the potential modder (or mod-user) can be better informed when things go south.
We all know the main problem with dictionaries is that they contain too many words, and not enough butterscotch sauce!

Derron

Simple solution:
- for each TSound you load you take its capacity
- for each image  the underlaying pixmap.width*pixmap.height*bitsPerPixel[pixmap.format]
- ...
- for each stuff you load you add 20% "on top"

So loading 1gb of pixel data results in your variable saying "1.2 gb"
Have a hard limit of eg 1.5gb allowed.
Throw an "warning" once this is reached ("you load a lot of media, programme might crash because of exceeding memory limits").



Regarding file:
place the .c file next to your .bmx files ... no need to place them anywhere else.

The issue you might have is "MinGW" - you need it as you need to compile some C code. If you only use "bmx" and never recompiled modules, then you might be in trouble.

BUT ... if you use "bmk.exe" from NG (it should be compatible) you can use your own MinGW anyways - and even could get rid of the "lib" folder as it then uses the one of the "MinGWx86" you can just place inside your BlitzMax folder (so it uses a local MinGW not the one you might have installed globally on your system because you are coding in other stuff than BlitzMax too).



Regarding vanilla:
I started my TVTower in the early 2000s - moved to BlitzMax (legacy) in 2005 or so. I coded it in legacy (logically :D). In 2020 I cut last connections to legacy. Prior I only tried to be "compatible" with NG to have 64bit support.
Since then I use the new features to save on RAM and GC Stress (object creation vs structs - and such stuff). Making it run in NG had some hurdles of course. First of all "rtAudio" which I used worked only in 64bit of NG. But I do not use rtAudio.mod anymore as Brucey then added audio.soloud which has streamed audio support too (so streaming your music - to keep memory usage low instead of loading the whole audio blob which decoded takes 10MB+ per minute when talking about normal stereo music),
Instead of using "libXML" you can also use "mxml" (became default) which is lighter and often faster too.
... and some more stuff.


If the modules you use are NG compatible, then you most probably are able to convert your existing game code to NG in a weekend. Especially if it is just 43 files (dunno how big they are :D) - and even more easier if you wrote your code "superstrict" (as I did).


so : if your modules are NG compatible I would suggest to (sooner or later) copy the folder and try to make it work with NG (we can be of help here of course). Ask here for assistance or use the official blitzmax ng discord channel - plenty of BMX users are there (similar to here).


bye
Ron

Adam Novagen

Woah... So it just straight-up works, plug-and-play. I honestly didn't expect it to be that simple, I thought I would have to at least build something :)) Thanks a million!
MinGW is fortunately already installed and configured, I did that many years ago when I needed to get Brucey's GME module running. I actually have a .TXT file with instructions in case I have to set the environment up again (and I have done, a few times) and an installer for good ol' TDM GCC 4.7.1, and the final version of BLIde (it fits like a comfy old shoe ;D).

Regarding NG, my thoughts are this: after spending so long getting everything as rock-solid as possible (those six years of developing a "simple" pixel RPG had to go somewhere!) I'm not going to rock the boat at this point. What I am considering is attempting a fork once the game is done, so that if me faffing about with NG causes death & destruction on a galactic scale I still have a proper completed build of the legacy program ready to roll. For future projects - and my next major release will be in Unity anyway, I'm talking personal stuff here - I'll be moving to NG for sure.

I would also like to say... I am endlessly grateful for the attitude of this community, and the constant willingness of folks like you and many others to provide help and encouragement to less experienced developers like me. This game, and everything I'll be doing after, has only been made possible by well over a decade of forum kindness from the lot of you, without which I would probably never have figured out how to build a game like this! :D
We all know the main problem with dictionaries is that they contain too many words, and not enough butterscotch sauce!

statto

Quote from: Derron on March 17, 2021, 08:38:41
If it was only for the GCMemory you could always use GCMemAlloced().

Little side information:
I know it is not what you are looking for - but with NG you could open ub blitz.mod/blitz_gc.h and enable bbGCAllocCount.
this would allow you to bring this "extern" variable into bmx and to count what elements are created (and maintained) by the GC.
Also you could dump out what the GC was collecting - I used it to create a table exposing if I leaked objects (kept references accidentally)

https://github.com/TVTower/TVTower/blob/master/source/Dig/base.util.bmxng.objectcountmanager.bmx

---

On Linux you could execute some of the available commands to grep the amount of memory your processes (if running in threads etc) use. But seems you ask for Win32. Aren't there some Win32-api commands available for your desired information?


Edit:
https://stackoverflow.com/questions/63166/how-to-determine-cpu-and-memory-consumption-from-inside-a-process

There is a example on how to retrieve total virtual memory - and also how to retrieve virtual memory used by the process


As you only need this particular information you might just copy the example code in a "getmemory.c" file (the "logic" wrapped inside a function returning the value) which you include in your application (inside the "extern" part).



bye
Ron

Sorry to bump an old thread, but my program had a very large memory leak that I had spent hours trying to find. Adding base.util.bmxng.objectcountmanager.bmx into the code allowed me to find it within minutes. (It was even dumber than you'd think - two object fields which generated thousands of objects per main loop wasn't being cleared after every loop due to an If statement not triggering, so not even really a "leak" but rather just continual object accumulation.)

Huge thank you for that bit of code.

Derron