[bb] Better WriteFile() and ReadFile() etc. by Streaksy [ 1+ years ago ]

[BlitzBot]

Title : Better WriteFile() and ReadFile() etc.
Author : Streaksy
Posted : 1+ years ago

Description : This is great for handling large files.  Use it like you would the native file handling functions.  There are a few brand new ones chucked in aswell.

The reason it's faster (in theory), and in my opinion, neater, is that files are read and written in one go and the rest is done in memory, which is much faster.  When writing, the xCloseFile function is what dumps the preprepared data to the file.  When reading, the XReadFile function reads the whole file in one sweep and subsequent read-data functions just read the values from memory.  Using this method with large files when you're only reading a small part of the file is therefore probably inferior.

I'm pretty sure it's all in working order and the code should be accessible enough.  Hope it's useful.  =D
*** PUBLIC FUNCTIONS:

WRITING:
XWriteFile(filename$,encrypt=0,compress=0)
XWriteInt(filehandle,value)
XWriteByte(filehandle,value)
XWriteFloat(filehandle,value#)
XWriteString(filehandle,value$)
XWriteLine(filehandle,value$)
XWriteBytes(bank,filehandle,value,offset,count)
XWriteImage(filehandle,image)
XWriteTexture(filehandle,texture,write_alpha_channel)
XWriteBank(filehandle,bank)
READING:

XReadFile(filename$,encrypt=0,compress=0)
XReadInt(filehandle)
XReadByte(filehandle)
XReadFloat#(filehandle)
XReadString$(filehandle)
XReadLine$(filehandle)
XReadBytes(Bank,filehandle,value,offset,count)
XReadImage(filehandle)
XReadTexture(filehandle,Read_alpha_channel)
XReadBank(filehandle,Bank)
UNIVERSAL:
XCloseFile(filehandle)
XEof(filehandle)
*** TOKENS:

You can define tokens which are strings that will be stored in compressed files as a few bytes,  It is useful for common words etc. that needlessly take up space in files.  You can safely ignore this whole system if it makes no sense to you or just isn't that useful for your purposes.

XAddToken(string$)

Code :

Code (blitzbasic) Select Expand

Global XMashKey$="This is the encryption key.  Make it as long and as random and as varied as you can, but make sure you use the same key to load files with which they were saved."



Global xDefaultFileSize=1000 ;how big a bank is when first created by xWriteFile()
Global bankstep=1500 ;how much is added to bank sizes when they get too small to contain the data (plus the size of the extra data)
Global maxfilehandles=500
Dim filehandle(maxfilehandles)
Dim filebank(maxfilehandles)
Dim filefn$(maxfilehandles)
Dim FileSiz(maxfilehandles)
Dim FileEnc(maxfilehandles)
Dim FileCmp(maxfilehandles)
Dim fileoffset(maxfilehandles)
Dim Filetyp(maxfilehandles) ;1 for read, 2 for write, 0 for closed, 3 for random (but i cant be arsed coding random)

Global XTokenID$=Chr(255) ;strings are tokenised when written to a file!
Global xTokens,MaxxTokens=255   ;long/common words can be replaced with a token!  Use XAddToken(word$) To define them
Dim xToken$(maxxTokens)




;DEMO
pic=LoadImage("C:Blitz3DProjectsdexter's test.png")
fil=xwritefile("moo.txt")
xwriteimage fil,pic
xclosefile fil

fil=xreadfile("moo.txt")
pic2=xreadimage(fil)
xclosefile fil

DrawImage pic2,0,0
WaitKey:DeleteFile "moo.txt":End




Function XSetKey(k$)
XMashKey$=k
End Function


Function XReadFile(fn$,enc=0,cmp=0)
sz=FileSize(fn)
fil=ReadFile(fn)
If fil=0 Then RuntimeError "Couldn't read from "+fn+"."
For t=1 To maxfilehandles
If filetyp(t)=0 Then
filesiz(t)=sz ;for XEof() etc
fileenc(t)=enc
filecmp(t)=cmp
fileoffset(t)=0
filetyp(t)=1
filebank(t)=CreateBank(sz)
filefn(t)=fn
ReadBytes filebank(t),fil,0,sz
CloseFile fil
If enc Then xmash filebank(t),sz
If cmp Then xdecompress filebank(t),sz
Return t
EndIf
Next
RuntimeError "Too many files open!"
End Function


Function XWriteFile(fn$,enc=0,cmp=0)
For t=1 To maxfilehandles
If filetyp(t)=0 Then
fileenc(t)=enc
filecmp(t)=cmp
fileoffset(t)=0
filetyp(t)=2
filebank(t)=CreateBank(xDefaultFileSize)
filefn(t)=fn
Return t
EndIf
Next
RuntimeError "Too many files open!"
End Function



Function XCloseFile(hnd)
If filetyp(hnd)=2 Then   ;closing a write handle, so write the whole lot then kill the bank
If filecmp(hnd) Then xcompress filebank(hnd),filesiz(hnd):filesiz(hnd)=BankSize(filebank(hnd))
If fileenc(hnd) Then xmash filebank(hnd),filesiz(hnd)
fil=WriteFile(filefn(hnd)):If fil=0 Then RuntimeError "Couldn't write to "+filefn(hnd)+"."
WriteBytes filebank(hnd),fil,0,filesiz(hnd)
CloseFile fil
FreeBank filebank(hnd)
filetyp(hnd)=0
EndIf
If filetyp(hnd)=1 Then   ;closing a read handle, so just kill the bank
FreeBank filebank(hnd)
filetyp(hnd)=0
EndIf
End Function


Function XEof(hnd)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use closed file handle!"
If FileTyp(hnd)<>1 Then RuntimeError "Can't check end-of-file for a file being written!"
If fileoffset(hnd)=>FileSiz(hnd) Then Return True
End Function


Function XWriteInt(hnd,val)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
offset=fileoffset(hnd):bnk=filebank(hnd)
If offset+4>filesiz(hnd) Then filesiz(hnd)=offset+4
If BankSize(bnk)<filesiz(hnd) Then ResizeBank bnk,filesiz(hnd)+bankstep
PokeInt(bnk,offset,val):fileoffset(hnd)=fileoffset(hnd)+4
End Function


Function XWriteFloat(hnd,val#)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
offset=fileoffset(hnd):bnk=filebank(hnd)
If offset+4>filesiz(hnd) Then filesiz(hnd)=offset+4
If BankSize(bnk)<filesiz(hnd) Then ResizeBank bnk,filesiz(hnd)+bankstep
PokeFloat(bnk,offset,val):fileoffset(hnd)=fileoffset(hnd)+4
End Function


Function XWriteByte(hnd,val)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
offset=fileoffset(hnd)
bnk=filebank(hnd)
If offset+1>filesiz(hnd) Then filesiz(hnd)=offset+1
If BankSize(bnk)<filesiz(hnd) Then ResizeBank bnk,filesiz(hnd)+bankstep
PokeByte(bnk,offset,val)
fileoffset(hnd)=fileoffset(hnd)+1
End Function


Function XWriteString(hnd,val$,tokenise=1)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
If tokenise Then If xtokens>0 Then val=xtokenise(val)
XWriteInt hnd,Len(val)
For t=1 To Len(val)
XWriteByte hnd,Asc(Mid(val,t,1))
Next
End Function


Function XWriteLine(hnd,val$,tokenise=1)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
If tokenise Then If xtokens>0 Then val=xtokenise(val)
For t=1 To Len(val)
XWriteByte hnd,Asc(Mid(val,t,1))
Next
XWriteByte hnd,13
XWriteByte hnd,10
End Function


Function XWriteBytes(bnk2,hnd,ofs,cnt)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>2 And filetyp(hnd)<>3 Then RuntimeError "Can't write to a non-writey file handle!"
offset=fileoffset(hnd)
bnk=filebank(hnd)
If offset+cnt>filesiz(hnd) Then filesiz(hnd)=offset+cnt
If BankSize(bnk)<filesiz(hnd) Then ResizeBank bnk,filesiz(hnd)+bankstep
CopyBank bnk2,ofs,bnk,offset,cnt
fileoffset(hnd)=fileoffset(hnd)+cnt
End Function






Function XReadByte(hnd)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
val=PeekByte(filebank(hnd),fileoffset(hnd))
fileoffset(hnd)=fileoffset(hnd)+1
Return val
End Function


Function XReadInt(hnd)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
val=PeekInt(filebank(hnd),fileoffset(hnd))
fileoffset(hnd)=fileoffset(hnd)+4
Return val
End Function


Function XReadFloat#(hnd)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
val#=PeekFloat(filebank(hnd),fileoffset(hnd))
fileoffset(hnd)=fileoffset(hnd)+4
Return val
End Function


Function XReadString$(hnd,tokenise=0)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
ln=XReadInt(hnd)
For p=1 To ln
val$=val$+Chr(XReadByte(hnd))
Next
If tokenise Then If xtokens>0 Then val=xdetokenise(val)
Return val
End Function


Function XReadLine$(hnd,tokenise=0)
If xeof(hnd) Then Return ""
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
For p=1 To 32000
If fileoffset(hnd)=>filesiz(hnd) Then Return val$;val$=val+Chr(13)+Chr(10):nored=1;RuntimeError "ReadLine timeout - end of file reached and no 13,10!"
If nored=0 Then val$=val$+Chr(XReadByte(hnd))
If Len(val)>2 Then
If Mid(val,p-1,1)=Chr(13) And Mid(val,p,1)=Chr(10) Then
val=Left(val,Len(val)-2)
If tokenise Then If xtokens>0 Then val=xdetokenise(val)
Return val
EndIf
EndIf
Next
RuntimeError "ReadLine timeout - no 13,10 found!"
End Function


Function XReadBytes(bnk2,hnd,ofs,cnt)
If hnd=0 Or hnd>maxfilehandles Then RuntimeError "Invalid file handle."
If filetyp(hnd)=0 Then RuntimeError "Can't use a closed handle file!"
If filetyp(hnd)<>1 And filetyp(hnd)<>3 Then RuntimeError "Can't read from a non-ready file handle!"
offset=fileoffset(hnd)
bnk=filebank(hnd)
CopyBank bnk,offset,bnk2,ofs,cnt
fileoffset(hnd)=fileoffset(hnd)+cnt
End Function


Function XMash(bnk,sz) ;for the reading/writing of encrypted files using mah xfile system... it gets called by XReadFile and XCloseFile
lk=Len(mashkey):kat=1
SeedRnd sz
b3=Rand(0,255)
For t=0 To sz-1
b1=PeekByte(bnk,t)
b3=b3+3:If b3>255 Then b3=b3-256
tik=tik+1:If tik=12 Then tik=0:b3=b3+Rand(3,121):If b3>255 Then b3=b3-256
b2=Asc(Mid(mashkey,kat,1)) Xor b3
PokeByte bnk,t,b1 Xor b2
kat=kat+1:If kat>lk Then kat=1
Next
End Function








Function xCompress(bnk,szz,cmp=0) ;does some RLE on the pesky zeroes
bnk2=CreateBank(szz*1.5)
t=0:t2=0
PokeByte bnk2,t2,cmp:t2=t2+1
Repeat
b=PeekByte(bnk,t):t=t+1
.rec
PokeByte bnk2,t2,b:t2=t2+1
If b<=cmp Then
If t=szz Then
PokeByte bnk2,t2,b:Goto donec
Else
cnt=1
For k=1 To 256
bb=PeekByte(bnk,t):t=t+1
If bb=b And t=szz Then ;end of bank, but last char was in the RLE chunk
cnt=cnt+1;tot up that final char then write the count
PokeByte bnk2,t2,cnt-1:t2=t2+1
Goto donec
EndIf
If bb<>b And t=szz Then ;end of bank, but last char wasnt in the RLE chunk
PokeByte bnk2,t2,cnt-1:t2=t2+1 ;write the count to conclude the RLE
PokeByte bnk2,t2,bb:t2=t2+1:If bb=<cmp Then PokeByte bnk2,t2,0:t2=t2+1 ;write the new byte, plus a 0 (+1) if it needs a RLE count
Goto donec
EndIf
If bb=b Then ;found another peice of the RLE chunk
cnt=cnt+1;tot it up
EndIf
If bb<>b Then ;past the RLE chunk
PokeByte bnk2,t2,cnt-1:t2=t2+1 ;write how many it had found
b=bb:Goto rec ;restart the byte check sequence on the new byte
EndIf
If k=256 Then
PokeByte bnk2,t2,255:t2=t2+1
Goto donebatch
EndIf
Next
.donebatch
EndIf
EndIf
.donec
Until t=szz
ResizeBank bnk,t2:CopyBank bnk2,0,bnk,0,t2:FreeBank bnk2
End Function


Function xDecompress(bnk,szz)
bnk2=CreateBank(szz*1.5)
t=0:t2=0
cmp=PeekByte(bnk,t2):t=t+1
Repeat
b=PeekByte(bnk,t):t=t+1
PokeByte bnk2,t2,b:t2=t2+1:If t2+10>BankSize(bnk2) Then ResizeBank bnk2,BankSize(bnk2)+200
If b=<cmp Then
cnt=PeekByte(bnk,t):t=t+1
For m=1 To cnt
PokeByte bnk2,t2,b:t2=t2+1::If t2+10>BankSize(bnk2) Then ResizeBank bnk2,BankSize(bnk2)+200
Next
EndIf
Until t=szz
ResizeBank bnk,t2:CopyBank bnk2,0,bnk,0,t2:FreeBank bnk2
End Function


Function xTokenise$(t$)
lll=Len(t)
For tt=1 To xtokens
If lll=>Len(xtoken(tt)) Then t=Replace(t,xtoken(tt),xtokenid+Chr(tt))
Next
Return t
End Function

Function xDeTokenise$(t$)
If Len(t)< Len(xtokenid)+1 Then Return t
For tt=1 To xtokens
t=Replace(t,xtokenid+Chr(tt),xtoken(tt))
Next
Return t
End Function

Function XAddToken(t$)
If Len(t)<(Len(xtokenid)+1) Then RuntimeError "Pointless token: "+t+" (it's so short that the token would be larger!)"
If Len(t)=(Len(xtokenid)+1) Then RuntimeError "Pointless token: "+t+" (it's so short that the token would be the same size!)"
If xtokens=>maxxtokens Then RuntimeError "Too many tokens: "+t
xtokens=xtokens+1
xtoken(xtokens)=t
End Function

















Function XWriteImage(f,img)
XWriteInt f,ImageWidth(img)
XWriteInt f,ImageHeight(img)
LockBuffer ImageBuffer(img)
For x=0 To ImageWidth(img)-1
For y=0 To ImageHeight(img)-1
hue=ReadPixelFast(x,y,ImageBuffer(img))
r=(hue And $00FF0000) Shr 16:XWriteByte f,r
g=(hue And $0000FF00) Shr 8:XWriteByte f,g
b=(hue And $000000FF):XWriteByte f,b
Next
Next
UnlockBuffer ImageBuffer(img)
End Function


Function XWriteTexture(f,tex,Alphah=0)
XWriteInt f,TextureWidth(tex)
XWriteInt f,TextureHeight(tex)
XWriteByte f,alphah
LockBuffer TextureBuffer(tex)
For x=0 To TextureWidth(tex)-1
For y=0 To TextureHeight(tex)-1
hue=ReadPixelFast(x,y,TextureBuffer(tex))
r=(hue And $00FF0000) Shr 16:XWriteByte f,r
g=(hue And $0000FF00) Shr 8:XWriteByte f,g
b=(hue And $000000FF):XWriteByte f,b
If alphah Then a=(hue And $FF000000) Shr 24:XWriteByte f,a
Next
Next
UnlockBuffer TextureBuffer(tex)
End Function


Function XWriteBank(f,bnk)
XWriteInt f,BankSize(bnk)
XWriteBytes bnk,f,0,BankSize(bnk)
End Function




Function XReadTexture(f,flags=9)
w=XReadInt(f)
h=XReadInt(f)
alphah=XReadByte(f)
tex=CreateTexture(w,h,flags)
LockBuffer TextureBuffer(tex)
For x=0 To TextureWidth(tex)-1
For y=0 To TextureHeight(tex)-1
r=XReadByte(f)
g=XReadByte(f)
b=XReadByte(f)
If alphah Then a=XReadByte(f)
hue=R Shl 16 Or G Shl 8 Or B
WritePixelFast x,y,hue,TextureBuffer(tex)
Next
Next
UnlockBuffer TextureBuffer(tex)
Return tex
End Function



Function XReadImage(f,flags=9)
w=XReadInt(f)
h=XReadInt(f)
img=CreateImage(w,h)
LockBuffer ImageBuffer(img)
For x=0 To ImageWidth(img)-1
For y=0 To ImageHeight(img)-1
r=XReadByte(f)
g=XReadByte(f)
b=XReadByte(f)
hue=R Shl 16 Or G Shl 8 Or B
WritePixelFast x,y,hue,ImageBuffer(img)
Next
Next
UnlockBuffer ImageBuffer(img)
Return img
End Function

Function XReadBank(f)
siz=XReadInt(f)
bnk=CreateBank(siz)
XReadBytes bnk,f,0,siz
Return bnk
End Function

Comments :


Streaksy(Posted 1+ years ago)

 Forgot to mention something and it won't let me edit my post for some reason...The encryption and compression are seperately optional.  You can have both, neither, or either one.  Just use the same options when loading that you used for saving.  Compression uses the mashkey$ which can be set any time with xSetKey.The compression technique is a simple RLE and won't ALWAYS be suitable, but for map files etc with a lot of empty data it should save a lot of space.  Either way compression and encryption slows it down a tad.


Doktor Zeus(Posted 1+ years ago)

 tried slipping this in as an include, but it keeps telling me that an image is missing on the second line of the WriteImage function. In fact it won't even run as a standalone. Am I doing something wrong here? Do I need a more current version of BB3D than v1.98?


_PJ_(Posted 1+ years ago)

 As postd above, the code auses this image file

Code Select Expand

pic=LoadImage("C:Blitz3DProjectsdexter's test.png")Make sure you change this path to your own valid image file.


Doktor Zeus(Posted 1+ years ago)

 Thanks. Big post, missed the most important bit. Hope I can work out how to get this working, as my own image-to-stream saving code is woefully inefficient!


Doktor Zeus(Posted 1+ years ago)

 Hm, Well, I got it working OK, but I'm finding odd results when I use compression to save a texture. For some weird reason I get three vertical red lines down the middle of the texture that actually pass behind parts of the image - at first I thought they were affecting the alpha map, then I remembered this was a bitmap and didn't have an alpha map. Furthermore, compression has practically no effect at all on the size of the file that is output, so there's little reason to turn it on anyway. Otherwise, it's a pretty neat package though I'll have to adapt it slightly to my own purposes I think, since I *do* want to read small chunks of a file.-Ash


Streaksy(Posted 1+ years ago)

 Yeh.. after using this a little more I've had strange results with RLE compression and I don't tend to bother with it anymore.  Since banks are used it won't take much adaptation to use a zlib dll...I'll leave this here 'cos the RLE is a small part of this code and I think people will still have a use for this.  The encryption is pretty good, I think. [/i]