LoadDir() is terrible on *nix platforms

Vectrex3000 · September 03, 2019, 10:15:57

Hi,

The built-in 'LoadDir()' function has some serious issues on *nix platforms (MacOS and Linux). It works perfectly fine on Windowns (x86 and x64).

Calling LoadDir() more than once, at any time in the application, will cause both MacOS and Linux applications to crash. In debug mode, as well as release, no specific error is thrown; Straight crash. A great majority of the time, LoadDir() causes MaxIDE crash leaving the application window in a hang.

This does not happen on Windows.

I tried this in a blank file, too. Var:string[] = LoadDir(dir) will crash if i call it again regardless if the second parameter is true/false.

Another interesting side effect is that i left my application running for about 41 minutes on Mac and Linux and they both crashed at the same time. I left them running again... crashed at the same time. I started them at the same time.

They didn't do this until i implemented loaddir() in my bootstrap procedure. Commenting out this section prevents the applications from crashing.

I left them running and they didn't crash. I ran them both and timed it.. they always crash in the 35 - 40 minute range.

Yes, this happens when i have a while statement in a blank file too. I timed everything and wasted hours doing the same tests..

LoadDir() is broken on *nix. Severely broken.

I'm using the Latest BMX NG 0.105... i just upgraded the other day from 0.98.

Derron · September 03, 2019, 10:41:18

I use LoadDir() and cannot replicate it for now (Linux and Windows).

Can you create a (non)working example ?

do you iterate over the "variable:string[]" ? And what are you doing there. Are you doing recursion there? Maybe you override the array while iterating over it ...

bye
Ron

Derron · September 03, 2019, 11:03:03

Code (BlitzMax) Select


SuperStrict

Local startDir:String = "/usr"
Local absoluteUri:String
For Local s1:String = EachIn LoadDir(startDir)
	absoluteUri = startDir + "/" + s1
	Print absoluteUri
	If FileType(absoluteUri) = FILETYPE_DIR
		For Local s2:String = EachIn LoadDir(absoluteUri)
			Print "  " + absoluteUri + "/" + s2
		Next
	EndIf
Next

(non recursive code - just to show that it calls LoadDir multiple times)

output:

Code Select


Building untitled1
[ 23%] Processing:untitled1.bmx
[ 85%] Compiling:untitled1.bmx.gui.release.linux.x64.c
[100%] Linking:untitled1
Executing:untitled1
/usr/include
  /usr/include/tar.h
  /usr/include/kadm5
  /usr/include/mircommon
  /usr/include/uchar.h
  /usr/include/error.h
  /usr/include/libmodplug
  /usr/include/freetype2
  /usr/include/ifaddrs.h
  /usr/include/rpc
  /usr/include/ldap_cdefs.h
  /usr/include/avahi-common
  /usr/include/netrom
  /usr/include/pngconf.h
  /usr/include/fstab.h
  /usr/include/gdk-pixbuf-2.0
  /usr/include/xen
  /usr/include/sndio.h
  /usr/include/sqlite3.h
  /usr/include/gail-1.0
  /usr/include/X11
[...]
  /usr/include/python3.5
/usr/lib32
  /usr/lib32/libubsan.so.0.0.0
  /usr/lib32/libgdk_pixbuf-2.0.so
  /usr/lib32/libcrypt.a
[...]

bye
Ron

Brucey · September 03, 2019, 17:11:47

Hallo,

I am not aware of any issues with LoadDir.

I am currently running this example on Ubuntu Linux 64-bit, against a dir of 15 entries. :

Code Select


SuperStrict

Framework brl.standardio
Import brl.filesystem

Const total:Int = 500000

Local count:Int
For Local i:Int = 0 Until total
	Local val:String[] = LoadDir(".")
	For Local s:String = EachIn val
		count :+ 1
	Next
	Delay(5)
	If i Mod 1000 = 0 Then
		Print i + " : " + count + "  (" + (total - i ) + ")"
	End If
Next

Some example output from the run ..

Code Select


47000 : 705015  (453000)
48000 : 720015  (452000)
49000 : 735015  (451000)
50000 : 750015  (450000)
51000 : 765015  (449000)
52000 : 780015  (448000)
53000 : 795015  (447000)
54000 : 810015  (446000)
55000 : 825015  (445000)
56000 : 840015  (444000)

First value is the number of calls to LoadDir. The next is the number of file names iterated over. The Final number is the remaining count of calls to go.

I'd guess LoadDir is not the issue.

I suggest running a crashing example under gdb on Linux, and get a backtrace from that - which would help to indicate where exactly the program was crashing.
You can use the "-gdb" developer option to build with gdb friendly debugging information.

Vectrex3000 · September 04, 2019, 07:03:18

Quote from: Derron on September 03, 2019, 11:03:03
Code (BlitzMax) Select Expand
SuperStrict Local startDir:String = "/usr" Local absoluteUri:String For Local s1:String = EachIn LoadDir(startDir) absoluteUri = startDir + "/" + s1 Print absoluteUri If FileType(absoluteUri) = FILETYPE_DIR For Local s2:String = EachIn LoadDir(absoluteUri) Print " " + absoluteUri + "/" + s2 Next EndIf Next
(non recursive code - just to show that it calls LoadDir multiple times)

output:
Code Select Expand
Building untitled1 [ 23%] Processing:untitled1.bmx [ 85%] Compiling:untitled1.bmx.gui.release.linux.x64.c [100%] Linking:untitled1 Executing:untitled1 /usr/include /usr/include/tar.h /usr/include/kadm5 /usr/include/mircommon /usr/include/uchar.h /usr/include/error.h /usr/include/libmodplug /usr/include/freetype2 /usr/include/ifaddrs.h /usr/include/rpc /usr/include/ldap_cdefs.h /usr/include/avahi-common /usr/include/netrom /usr/include/pngconf.h /usr/include/fstab.h /usr/include/gdk-pixbuf-2.0 /usr/include/xen /usr/include/sndio.h /usr/include/sqlite3.h /usr/include/gail-1.0 /usr/include/X11 [...] /usr/include/python3.5 /usr/lib32 /usr/lib32/libubsan.so.0.0.0 /usr/lib32/libgdk_pixbuf-2.0.so /usr/lib32/libcrypt.a [...]

bye
Ron

i'm using a resize-able array that works flawlessly on windows.

All i do is call loaddir() into a string array. However, along with this, i also check that no directories are included.
I store the number files in an INT
I have a FOR loop that parses every entry containing a specific filename into another string array and i count those
that's it.

I only check for this every fps*80 (i use 25) ticks since in unnecessary for me to constantly do this. The next time this sequence it called, it crashes.
It's not an array index overflow. I can tell it's loaddir on my end because if i remove loaddir() and just spoof the contents of my array with the
contents of another resizeable string array i have, it works just fine. It's just on Mac and Linux that this happened; Windows is fine.

The windows edition of my program runs just fine without any issues. 40ish minutes in, on BOTH mac and linux, the application will crash.
Removing loaddir() related stuff prevents this. When loaddir() is forced to be called a second time, by shortening ticks till called, mac and linux crash.

Derron · September 04, 2019, 07:10:27

Do you have any unicode characters in the name of your directories' entries? So filenames with utf8-characters or so? Maybe this leads to issues?

Could you post the part doing the loaddir() + passing to string array?
It should also crash if you remove every interaction with your program (so calling custom functions etc). It should crash with just LoadDir + your array accesses. And if it does then please try to upload this piece of code so people ("we") could have a look.

@ resizeable array
so you have
local myfiles:string[]
and add them via "myfiles :+ [filename]" or do you manual resizing via "myfiles = myfiles[.. myfiles.length*2]; ... ; ...myfiles = myfiles[.. fileCount]" ?

bye
Ron

Brucey · September 04, 2019, 20:31:40

Okay, so you aren't just calling LoadDir(), you are doing a lot of other things - the bug that's causing the problem would be in the other things then.

Since I don't have access to your source, I can't really help any further with your problem.
However, since I do have the source for LoadDir, I am confident that the LoadDir code is okay. (you also have access to the source for that, should you wish to look further into that yourself).

Without seeing your problem code, we are just going to go through a pointless guessing game of what may be wrong.
Feel free to PM me, or send me your code for me to look at privately if you'd like some further help with this.