[BMX] DataPak Manager With Directory Structure

Started by Filax, April 09, 2025, 09:38:44

Previous topic - Next topic
Print
Go Down Pages1
User actions

Filax

April 09, 2025, 09:38:44 Last Edit: April 09, 2025, 10:33:34 by Filax
The most shorter package manager i ever made!  :))

It allow you to store any file / text string / value inside a compressed zip package. You can also package a directory
in one time and you keep the structure inside you code, example :

Code: BASIC
Directory/
├── Node_Icon_Branch_A.png
├── Node_Icon_CheckList.png
└── Test/
    └── Node_Icon_App.png

The program scan the sub folders. It's a prototype i'm sure there is some bugs! :)  One cool stuff is :
You don't load the full package in memory! It read only what it need...

But i test it with more than 500 jpeg inside a directory and i can draw my images without problem.
The way is drawed! If you improve the code, share it! :)

NOTE: Edited, now include sound / float and Int

Code: BASIC
' ------------------------------------------------
' Name : Data Package System V0.4
' Date : (C)2025
' Site:https://github.com/BlackCreepyCat
' -----------------------------------------------
SuperStrict

' -------------------
' Required modules
' -------------------
Import brl.max2d
Import brl.pixmap
Import brl.standardio
Import brl.filesystem
Import brl.bank
Import brl.audio  ' Pour les sons
Import Archive.ZLib

' ------------------------------------
' Constants for data types
' ------------------------------------
Const TYPE_FILE:Int = 0    ' Raw file
Const TYPE_TEXT:Int = 1    ' Text
Const TYPE_IMAGE:Int = 2   ' Image (Pixmap)
Const TYPE_STRING:Int = 3  ' String (comme une variable)
Const TYPE_INT:Int = 4     ' Integer
Const TYPE_FLOAT:Int = 5   ' Float
Const TYPE_SOUND:Int = 6   ' Sound

Const HEADER_STRING:String = "PAK1.0 (C)2025 By CreepyCat"

' Structure pour un élément dans l'index
Type TPakEntry
    Field dataType:Int      ' Type of data
    Field name:String       ' Name of the element (with relative path)
    Field offset:Long       ' Position in the file where compressed data starts
    Field fullSize:ULongInt ' Uncompressed size of the data
    Field packedSize:ULongInt ' Compressed size of the data
    Field packedData:TBank  ' Temporary storage for compressed data
End Type

' ----------------------------
' Class to manage the package
' ----------------------------
Type TPakManager
    Global entries:TList = CreateList()  ' List to store all entries

    ' -------------------------
    ' Add a file to the package
    ' -------------------------
    Function AddFile(filename:String, basePath:String)
        If FileType(filename) <> 1 Then Return
        Local file:TStream = ReadFile(filename)
        Local fullSize:ULongInt = FileSize(filename)
        If fullSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "File too large for TBank.Read: " + filename
        If fullSize > $7FFFFFFF Then RuntimeError "File too large for CreateBank: " + filename
        
        Local bank:TBank = CreateBank(Int(fullSize))
        bank.Read(file, 0, Long(fullSize))
        CloseFile(file)

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_FILE
        entry.name = filename.Replace(basePath + "/", "")
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added file: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' -----------------------
    ' Add text to the package
    ' -----------------------
    Function AddText(text:String, name:String)
        Local fullSize:ULongInt = text.Length
        If fullSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Text too large for TBank.Read: " + name
        If fullSize > $7FFFFFFF Then RuntimeError "Text too large for CreateBank: " + name
        
        Local bank:TBank = CreateBank(Int(fullSize))
        Local textBuf:Byte Ptr = text.ToCString()
        MemCopy(bank.Buf(), textBuf, Size_T(fullSize))
        MemFree textBuf

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_TEXT
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added text: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ---------------------------
    ' Add an image to the package
    ' ---------------------------
    Function AddImage(pixmap:TPixmap, name:String)
        Local bufferSize:ULongInt = pixmap.Width * pixmap.Height * BytesPerPixel[pixmap.Format]
        Local fullSize:ULongInt = 12 + bufferSize
        If fullSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Image too large for TBank.Read: " + name
        If fullSize > $7FFFFFFF Then RuntimeError "Image too large for CreateBank: " + name
        
        Local bank:TBank = CreateBank(Int(fullSize))
        bank.PokeInt(0, pixmap.Width)
        bank.PokeInt(4, pixmap.Height)
        bank.PokeInt(8, pixmap.Format)
        MemCopy(bank.Buf() + 12, pixmap.pixels, Size_T(bufferSize))

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_IMAGE
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added image: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ---------------------------
    ' Add a string to the package
    ' ---------------------------
    Function AddString(value:String, name:String)
        Local fullSize:ULongInt = value.Length
        If fullSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "String too large for TBank.Read: " + name
        If fullSize > $7FFFFFFF Then RuntimeError "String too large for CreateBank: " + name
        
        Local bank:TBank = CreateBank(Int(fullSize))
        Local valueBuf:Byte Ptr = value.ToCString()
        MemCopy(bank.Buf(), valueBuf, Size_T(fullSize))
        MemFree valueBuf

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_STRING
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added string: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ---------------------------
    ' Add an int to the package
    ' ---------------------------
    Function AddInt(value:Int, name:String)
        Local fullSize:ULongInt = 4
        Local bank:TBank = CreateBank(Int(fullSize))
        bank.PokeInt(0, value)

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_INT
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added int: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ---------------------------
    ' Add a float to the package
    ' ---------------------------
    Function AddFloat(value:Float, name:String)
        Local fullSize:ULongInt = 4
        Local bank:TBank = CreateBank(Int(fullSize))
        bank.PokeFloat(0, value)

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_FLOAT
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added float: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ---------------------------
    ' Add a sound to the package
    ' ---------------------------
    Function AddSound(filename:String, name:String)
        If FileType(filename) <> 1 Then RuntimeError "Sound file not found: " + filename
        Local file:TStream = ReadFile(filename)
        Local fullSize:ULongInt = FileSize(filename)
        If fullSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Sound too large for TBank.Read: " + filename
        If fullSize > $7FFFFFFF Then RuntimeError "Sound too large for CreateBank: " + filename
        
        Local bank:TBank = CreateBank(Int(fullSize))
        bank.Read(file, 0, Long(fullSize))
        CloseFile(file)

        Local entry:TPakEntry = New TPakEntry
        entry.dataType = TYPE_SOUND
        entry.name = name
        entry.fullSize = fullSize
        entry.packedData = Pak_CompressBank(bank)
        entry.packedSize = entry.packedData.Size()
        entries.AddLast(entry)

        Print "Added sound: " + entry.name + " (" + entry.fullSize + " -> " + entry.packedSize + " bytes)"
    End Function

    ' ------------------------------
    ' Add all files from a directory
    ' ------------------------------
    Function AddDirectory(dirPath:String, basePath:String = "")
        If basePath = "" Then basePath = dirPath
        
        Local dir:Byte Ptr = ReadDir(dirPath)
        If Not dir Then RuntimeError "Cannot open directory: " + dirPath

        Local file:String
        
        Repeat
            file = NextFile(dir)
            If file = "" Then Exit
            If file = "." Or file = ".." Then Continue
            
            Local fullPath:String = dirPath + "/" + file
            
            If FileType(fullPath) = 1 Then
                AddFile(fullPath, basePath)
            ElseIf FileType(fullPath) = 2 Then
                AddDirectory(fullPath, basePath)
            EndIf
        Forever
        
        CloseDir(dir)
        Print "Added all files from directory: " + dirPath
    End Function

    ' -----------------------
    ' Create the package file
    ' -----------------------
    Function CreatePak(filename:String)
        Local pakStream:TStream = WriteFile(filename)
        If Not pakStream Then RuntimeError "Cannot create package: " + filename

        pakStream.WriteLine(HEADER_STRING)
        pakStream.WriteInt(entries.Count())

        Local indexSize:Long = 0
        For Local entry:TPakEntry = EachIn entries
            indexSize :+ 4 + 4 + entry.name.Length + 8 + 8 + 8
        Next

        Local dataOffset:Long = pakStream.Pos() + indexSize
        Local indexOffset:Long = pakStream.Pos()
        
        For Local entry:TPakEntry = EachIn entries
            pakStream.WriteInt(entry.dataType)
            pakStream.WriteInt(entry.name.Length)
            pakStream.WriteString(entry.name)
            pakStream.WriteLong(0)
            pakStream.WriteLong(Long(entry.fullSize))
            pakStream.WriteLong(Long(entry.packedSize))
        Next

        Local currentOffset:Long = dataOffset
        For Local entry:TPakEntry = EachIn entries
            entry.offset = currentOffset
            pakStream.WriteBytes(entry.packedData.Buf(), Long(entry.packedSize))
            currentOffset :+ entry.packedSize
        Next

        SeekStream(pakStream, indexOffset)
        For Local entry:TPakEntry = EachIn entries
            pakStream.WriteInt(entry.dataType)
            pakStream.WriteInt(entry.name.Length)
            pakStream.WriteString(entry.name)
            pakStream.WriteLong(entry.offset)
            pakStream.WriteLong(Long(entry.fullSize))
            pakStream.WriteLong(Long(entry.packedSize))
        Next

        CloseFile(pakStream)
        Print "Package created: " + filename + " (" + FileSize(filename) + " bytes)"
    End Function

    ' ---------------------
    ' List package contents
    ' ---------------------
    Function ListPak(filename:String)
        Local pakStream:TStream = ReadFile(filename)
        If Not pakStream Then RuntimeError "Cannot open package: " + filename

        Local header:String = pakStream.ReadLine()
        If header <> HEADER_STRING Then RuntimeError "Invalid package format"

        Local numEntries:Int = pakStream.ReadInt()
        Print "Contents of package " + filename + " (" + numEntries + " elements):"
        
        For Local i:Int = 0 Until numEntries
            Local entry:TPakEntry = New TPakEntry
            entry.dataType = pakStream.ReadInt()
            Local nameLength:Int = pakStream.ReadInt()
            entry.name = pakStream.ReadString(nameLength)
            entry.offset = pakStream.ReadLong()
            entry.fullSize = pakStream.ReadLong()
            entry.packedSize = pakStream.ReadLong()
            Print " - " + entry.name + " (type=" + entry.dataType + ", full=" + entry.fullSize + ", packed=" + entry.packedSize + ", offset=" + entry.offset + ")"
        Next

        CloseFile(pakStream)
    End Function

    ' -----------------------------
    ' Load an item from the package
    ' -----------------------------
    Function LoadItem:Object(filename:String, itemName:String)
        Local pakStream:TStream = ReadFile(filename)
        If Not pakStream Then Return Null

        Local header:String = pakStream.ReadLine()
        If header <> HEADER_STRING Then RuntimeError "Invalid package format"

        Local numEntries:Int = pakStream.ReadInt()
        Local targetEntry:TPakEntry
        
        For Local i:Int = 0 Until numEntries
            Local entry:TPakEntry = New TPakEntry
            entry.dataType = pakStream.ReadInt()
            Local nameLength:Int = pakStream.ReadInt()
            entry.name = pakStream.ReadString(nameLength)
            entry.offset = pakStream.ReadLong()
            entry.fullSize = pakStream.ReadLong()
            entry.packedSize = pakStream.ReadLong()
            If entry.name = itemName Then targetEntry = entry ; Exit
        Next

        If Not targetEntry Then
            CloseFile(pakStream)
            Print "Item not found in package: " + itemName
            Return Null
        EndIf

        SeekStream(pakStream, targetEntry.offset)
        If targetEntry.packedSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Compressed data too large for TBank.Read: " + itemName
        If targetEntry.packedSize > $7FFFFFFF Then RuntimeError "Compressed data too large for CreateBank: " + itemName
        
        Local packedBank:TBank = CreateBank(Int(targetEntry.packedSize))
        packedBank.Read(pakStream, 0, Long(targetEntry.packedSize))
        Local bank:TBank = Pak_UncompressBank(packedBank)
        CloseFile(pakStream)

        Select targetEntry.dataType
            Case TYPE_FILE
                Local stream:TBankStream = CreateBankStream(bank)
                Return stream
            Case TYPE_TEXT
                Return String.FromBytes(bank.Buf(), bank.Size())
            Case TYPE_IMAGE
                Local width:Int = bank.PeekInt(0)
                Local height:Int = bank.PeekInt(4)
                Local format:Int = bank.PeekInt(8)
                If format < 1 Or format > 6 Then
                    Print "Invalid format for " + itemName + ": " + format
                    Return Null
                EndIf
                Local pixmap:TPixmap = CreatePixmap(width, height, format)
                If Not pixmap Then Return Null
                MemCopy(pixmap.pixels, bank.Buf() + 12, Size_T(bank.Size() - 12))
                Return LoadImage(pixmap)
            Case TYPE_STRING
                Return String.FromBytes(bank.Buf(), bank.Size())
            Case TYPE_INT
                Return String(bank.PeekInt(0))
            Case TYPE_FLOAT
                Return String(bank.PeekFloat(0))
            Case TYPE_SOUND
                Local stream:TBankStream = CreateBankStream(bank)
                SeekStream(stream, 0)  ' S'assurer que le curseur est au début
                Local sound:TSound = LoadSound(stream)
                If Not sound Then
                    Print "Failed to load sound from stream: " + itemName
                    ' Pour debug : sauvegarder le flux décompressé dans un fichier temporaire
                    Local tempFile:TStream = WriteFile("temp_" + itemName)
                    bank.Save(tempFile)
                    CloseFile(tempFile)
                    Print "Saved decompressed sound to temp_" + itemName + " for debugging"
                EndIf
                Return sound
        End Select

        Return Null
    End Function
End Type

' -----------------------
' Compress a bank of data
' -----------------------
Function Pak_CompressBank:TBank(bank:TBank)
    Local size:ULongInt = bank.Size()
    Local outSize:ULongInt = size + size / 10 + 32
    If outSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Estimated compressed size too large for compress2"
    If outSize > $7FFFFFFF Then RuntimeError "Estimated compressed size too large for CreateBank"
    
    Local out:TBank = TBank.Create(Int(outSize))
    compress2(out.Buf() + 8, outSize, bank.Buf(), size, 9)
    out.PokeLong(0, Long(size))
    out.Resize(Int(outSize + 8))
    
    Return out
End Function

' -------------------------
' Decompress a bank of data
' -------------------------
Function Pak_UncompressBank:TBank(bank:TBank)
    Local outSize:ULongInt = bank.PeekLong(0)
    If outSize > $7FFFFFFFFFFFFFFF:ULongInt Then RuntimeError "Decompressed size too large for uncompress"
    If outSize > $7FFFFFFF Then RuntimeError "Decompressed size too large for CreateBank"
    
    Local out:TBank = TBank.Create(Int(outSize))
    uncompress(out.Buf(), outSize, bank.Buf() + 8, ULongInt(bank.Size() - 8))
    Return out
End Function

' -----------------------------------------
' Main function to test the package manager
' -----------------------------------------
Function Main()
    Graphics(800, 600)

    ' -------------------------------
    ' Create and populate the package
    ' -------------------------------
    Local pak:TPakManager = New TPakManager
    pak.AddDirectory("directory")
    pak.AddText("Hello world", "Salutation.txt")
    
    Local pixmap:TPixmap = LoadPixmap("Node_Icon_Bluetooth.png")
    If pixmap Then
        pak.AddImage(pixmap, "image_directe")
    EndIf
    
    pak.AddString("CreepyCat", "PlayerName")
    pak.AddInt(42, "PlayerLevel")
    pak.AddFloat(3.14159, "PlayerSpeed")
    pak.AddSound("Directory/Dreamkid.wav", "TestSound")
    
    pak.CreatePak("test.pak")

    ' -------------
    ' List contents
    ' -------------
    TPakManager.ListPak("test.pak")

    ' ---------------------------
    ' Load items from the package
    ' ---------------------------
    Local fileStream:TBankStream = TBankStream(TPakManager.LoadItem("test.pak", "Test/Node_Icon_App.png"))
    Local pixmapFromPak:TPixmap
    If fileStream Then
        pixmapFromPak = LoadPixmap(fileStream)
    EndIf
    
    Local text:String = String(TPakManager.LoadItem("test.pak", "Salutation.txt"))
    Local image:TImage
    If pixmapFromPak Then
        image = LoadImage(pixmapFromPak)
    EndIf
    
    Local loadedString:String = String(TPakManager.LoadItem("test.pak", "PlayerName"))
    Local loadedInt:Int = Int(String(TPakManager.LoadItem("test.pak", "PlayerLevel")))
    Local loadedFloat:Float = Float(String(TPakManager.LoadItem("test.pak", "PlayerSpeed")))
    Local loadedSound:TSound = TSound(TPakManager.LoadItem("test.pak", "TestSound"))

    Print "Loaded text: " + text
    If image Then
        Print "Image loaded: OK"
    Else
        Print "Image loading error"
    EndIf
    Print "Loaded string: " + loadedString
    Print "Loaded int: " + loadedInt
    Print "Loaded float: " + loadedFloat
    
    If loadedSound Then
        Print "Sound loaded: OK"
    Else
        Print "Sound loading error"
    EndIf

    ' Jouer le son si chargé
    If loadedSound Then
        PlaySound(loadedSound)
    Else
        Print "Impossible de jouer le son"
    EndIf

    ' ------------
    ' Display loop
    ' ------------
    While Not KeyHit(KEY_ESCAPE)
        Cls
        DrawText("Text: " + text, 10, 10)
        If image Then DrawImage(image, 10, 50)
        DrawText("String: " + loadedString, 10, 100)
        DrawText("Int: " + loadedInt, 10, 120)
        DrawText("Float: " + loadedFloat, 10, 140)
        
        If loadedSound Then
            DrawText("Sound: Playing 'TestSound'", 10, 160)
        Else
            DrawText("Sound: Not loaded", 10, 160)
        EndIf
        
        DrawText("Press ESC to quit", 10, 580)
        Flip
    Wend

    EndGraphics()
End Function

Main()


Derron

#1
April 10, 2025, 07:46:21
You might want to check out "brl.io" for utilizing existing zips - it allows you to map zip files as virtual folders (even over existing folders - nice for modding)

Also there is archive.mod - which allows you to handle a lot of archives - including eg zips with zstd compression. The zstd compression is way faster than zip and has a good compression rate (I use it for compressing my savegames - as it is 3-4 MB vs 100 MB :P).


bye
Ron

Filax

#2
April 10, 2025, 07:55:15

Quote from: Derron on April 10, 2025, 07:46:21You might want to check out "brl.io" for utilizing existing zips - it allows you to map zip files as virtual folders (even over existing folders - nice for modding)
Also there is archive.mod - which allows you to handle a lot of archives - including eg zips with zstd compression. The zstd compression is way faster than zip and has a good compression rate (I use it for compressing my savegames - as it is 3-4 MB vs 100 MB :P).
bye
Ron

 "brl.io"? I'll explore it ! I must admit that I'm a little lost with the versions of blitzmax... I dont know what is the best updated!

Blitzmax NG / Blitzmax.org ? 

You never know where to download the latest version. Between blitzmax NG? Or the blitzmax that is on Blitzmax.org ?
we can't find our way around! Modules work on one... not on the other!

All my old sources crash 50% because of the modules that have changed either name or location! Bah module Cairo! (my heart bleeding)
do not work anymore, like 60% of the Bah modules.

It's a real mess...  :)) And the documentation about module! It's very hard to find something updated... Anyway thanks for the info!

 

Filax

#3
April 10, 2025, 08:56:14
Thanks for this nice informations about those module! The code is a bit more shorter now :) Thanks for this!

A simple example for beginners:

Code: BASIC
SuperStrict

Import Archive.Core
Import archive.zip
Import Crypto.Crypto
Import BRL.StandardIO ' For Print

' Function to create a ZIP file with AES-256 encrypted data
Function CreateAESProtectedZip(zipFileName:String, fileContent:String, key:TCryptoSecretBoxKey)
 Local archive:TWriteArchive = New TWriteArchive
 
 archive.SetFormat(EArchiveFormat.Zip)
 archive.SetFormatOption("compression", "store") ' No ZIP compression

 If archive.Open(zipFileName) <> ARCHIVE_OK Then
 Print "Failed to open archive: " + archive.ErrorString()
 archive.Free()
 Return
 End If

 ' Encrypt data using TCryptoSecretBox
 Local contentBytes:Byte Ptr = fileContent.ToUTF8String()
 Local contentSize:Size_T = GetBytePtrLength(contentBytes)
 Local encrypted:Byte[] = New Byte[contentSize + CRYPTO_SECRETBOX_HEADERBYTES]

 If Not TCryptoSecretBox.Encrypt(encrypted, Size_T(encrypted.Length), contentBytes, contentSize, 0, "mydata", key) Then
 Print "Encryption failed"
 archive.Free()
 Return
 End If

 ' Create an entry in the ZIP file
 Local entry:TArchiveEntry = New TArchiveEntry
 entry.SetPathname("data.enc") ' File extension indicates encryption
 entry.SetSize(encrypted.Length)
 entry.SetFileType(EArchiveFileType.File)

 If archive.Header(entry) <> ARCHIVE_OK Then
 Print "Header creation failed: " + archive.ErrorString()
 archive.Free()
 Return
 End If

 If archive.Data(encrypted, Size_T(encrypted.Length)) < 0 Then
 Print "Writing data failed: " + archive.ErrorString()
 archive.Free()
 Return
 End If

 If archive.FinishEntry() <> ARCHIVE_OK Then
 Print "Entry finalization failed: " + archive.ErrorString()
 archive.Free()
 Return
 End If

 If archive.Close() <> ARCHIVE_OK Then
 Print "Closing archive failed: " + archive.ErrorString()
 Else
 Print "Encrypted ZIP file created: " + zipFileName
 End If

 archive.Free()
End Function

' Function to get the size of a Byte Ptr
Function GetBytePtrLength:Size_T(bytePtr:Byte Ptr)
 Local length:Size_T = 0
 While bytePtr[length] <> 0
 length:+1
 Wend
 Return length
End Function

' Test
Local key:TCryptoSecretBoxKey = TCryptoSecretBox.KeyGen() ' Generate AES-256 key
CreateAESProtectedZip("protected.zip", "This is a test ZIP file.", key)
Print "Key (keep it safe!): " + key.ToString()





Filax

#4
April 10, 2025, 10:29:20
Another example:

Code: BASIC
' ------------------------------------------------
' Name : Data Package System V0.6
' Date : (C)2025 Thanks to Derron! :)
' Site:https://github.com/BlackCreepyCat
' -----------------------------------------------
SuperStrict

Import Archive.Core
Import archive.zip
Import Crypto.Crypto
Import BRL.FileSystem
Import BRL.StandardIO

' --------------------
' Constants Definition
' --------------------
' Define constants used for custom archive header
Const CUSTOM_HEADER:String = "MYARCHIVE" ' Magic identifier
Const CUSTOM_VERSION:Byte = 1 ' Format version

' ----------------
' Type Definition
' ----------------
' Define a type to manage archive operations with encryption
Type TArchiveManager
    Field key:TCryptoSecretBoxKey ' Encryption key
    Field useCustomFormat:Int = False ' True for custom format, False for ZIP
    
    ' -----------------------------------
    ' Method: Constructor
    ' -----------------------------------
    ' Initialize a new TArchiveManager instance with a passphrase and format choice
    Method New(passphrase:String, customFormat:Int = False)
        key = TCryptoSecretBox.KeyGen() ' Generate a unique encryption key
        useCustomFormat = customFormat ' Set the archive format preference
    End Method
    
    ' -----------------------------------
    ' Method: ArchiveDirectory
    ' -----------------------------------
    ' Store a directory into an archive based on format choice
    Method ArchiveDirectory:Int(dirPath:String, archivePath:String)

        If useCustomFormat Then
            Return ArchiveDirectoryCustom(dirPath, archivePath)
        Else
            Return ArchiveDirectoryZip(dirPath, archivePath)
        End If

    End Method
    
    ' -----------------------------------
    ' Method: ArchiveDirectoryZip
    ' -----------------------------------
    ' Internal method to create a ZIP archive from a directory
    Method ArchiveDirectoryZip:Int(dirPath:String, archivePath:String)

        ' -----------------------------------
        ' Check if directory exists
        ' -----------------------------------
        ' Ensure the input directory is valid before proceeding
        If FileType(dirPath) <> FILETYPE_DIR Then
            Print "Error: " + dirPath + " is not a valid directory"
            Return False
        End If

        Local archive:TWriteArchive = New TWriteArchive
        archive.SetFormat(EArchiveFormat.Zip)
        archive.SetFormatOption("compression", "store") ' No compression
        
        ' -----------------------------------
        ' Attempt to open archive for writing
        ' -----------------------------------
        ' Open the ZIP file for writing, exit if it fails
        If archive.Open(archivePath) <> ARCHIVE_OK Then
            Print "Failed to open archive: " + archive.ErrorString()
            archive.Free()
            Return False
        End If
        
        Local success:Int = True
        
        ' -----------------------------------
        ' Loop through directory files
        ' -----------------------------------
        ' Iterate over all files in the directory to add them to the archive
        For Local file:String = EachIn FileList(dirPath, True)
            ' Preserve "directory/" in the relative path for ZIP structure
            Local relativePath:String = dirPath + "/" + file[dirPath.Length + 1..]
            Local entry:TArchiveEntry = New TArchiveEntry
            entry.SetPathname(relativePath)
            entry.SetFileType(EArchiveFileType.File)
            
            Local content:Byte[] = LoadByteArray(file)

            ' -----------------------------------
            ' Check file loading
            ' -----------------------------------
            ' Verify that file content was loaded successfully
            If Not content Then
                Print "Failed to load file: " + file
                success = False
                Continue
            End If

            Local encrypted:Byte[] = EncryptData(content)

            ' -----------------------------------
            ' Check encryption
            ' -----------------------------------
            ' Ensure encryption of file content succeeded
            If Not encrypted Then
                Print "Failed to encrypt file: " + file
                success = False
                Continue
            End If

            entry.SetSize(encrypted.Length)
            
            ' -----------------------------------
            ' Write header to archive
            ' -----------------------------------
            ' Add entry header to the ZIP archive
            If archive.Header(entry) <> ARCHIVE_OK Then
                Print "Failed to write header for " + relativePath + " : " + archive.ErrorString()
                success = False
                Continue
            End If
            
            ' -----------------------------------
            ' Write encrypted data
            ' -----------------------------------
            ' Write the encrypted file content to the archive
            If archive.Data(encrypted, Size_T(encrypted.Length)) < 0 Then
                Print "Failed to write data for " + relativePath + " : " + archive.ErrorString()
                success = False
                Continue
            End If
            
            ' -----------------------------------
            ' Finalize entry
            ' -----------------------------------
            ' Complete the current entry in the archive
            If archive.FinishEntry() <> ARCHIVE_OK Then
                Print "Failed to finalize entry for " + relativePath + " : " + archive.ErrorString()
                success = False
            End If

        Next
        
        ' -----------------------------------
        ' Close the archive
        ' -----------------------------------
        ' Finalize and close the ZIP archive
        If archive.Close() <> ARCHIVE_OK Then
            Print "Failed to close archive: " + archive.ErrorString()
            success = False
        Else
            Print "ZIP archive created: " + archivePath
        End If
        
        archive.Free()
        Return success
    End Method
    
    ' -----------------------------------
    ' Method: ArchiveDirectoryCustom
    ' -----------------------------------
    ' Internal method to create a custom format archive
    Method ArchiveDirectoryCustom:Int(dirPath:String, archivePath:String)

        ' -----------------------------------
        ' Validate directory
        ' -----------------------------------
        ' Check if the input directory exists
        If FileType(dirPath) <> FILETYPE_DIR Then
            Print "Error: " + dirPath + " is not a valid directory"
            Return False
        End If
        
        Local stream:TStream = WriteFile(archivePath)

        ' -----------------------------------
        ' Open output file
        ' -----------------------------------
        ' Open the file for writing custom archive data
        If Not stream Then
            Print "Failed to open file"
            Return False
        End If
        
        ' Write custom header
        stream.WriteBytes(CUSTOM_HEADER.ToUTF8String(), CUSTOM_HEADER.Length)
        stream.WriteByte(CUSTOM_VERSION)
        
        ' Store the number of files
        Local files:String[] = FileList(dirPath, True)
        stream.WriteInt(files.Length)
        
        ' -----------------------------------
        ' Loop through files for custom archive
        ' -----------------------------------
        ' Add each file to the custom archive with its path and encrypted data
        For Local file:String = EachIn files
            Local relativePath:String = dirPath + "/" + file[dirPath.Length + 1..]
            Local content:Byte[] = LoadByteArray(file)

            If Not content Then
                Print "Failed to load file: " + file
                Continue
            End If

            Local encrypted:Byte[] = EncryptData(content)

            If Not encrypted Then
                Print "Failed to encrypt file: " + file
                Continue
            End If
            
            ' Write path length and path
            Local pathBytes:Byte Ptr = relativePath.ToUTF8String()
            Local pathLen:Size_T = GetBytePtrLength(pathBytes)
            stream.WriteInt(Int(pathLen))
            stream.WriteBytes(pathBytes, pathLen)
            
            ' Write size and encrypted data
            stream.WriteInt(encrypted.Length)
            stream.WriteBytes(encrypted, encrypted.Length)

        Next
        
        stream.Close()
        Print "Custom archive created: " + archivePath
        Return True
    End Method
    
    ' -----------------------------------
    ' Method: ReadFile
    ' -----------------------------------
    ' Read a specific file from an archive based on format
    Method ReadFile:String(archivePath:String, filePath:String)

        If useCustomFormat Then
            Return ReadFileCustom(archivePath, filePath)
        Else
            Return ReadFileZip(archivePath, filePath)
        End If

    End Method
    
    ' -----------------------------------
    ' Method: ReadFileZip
    ' -----------------------------------
    ' Internal method to read a file from a ZIP archive
    Method ReadFileZip:String(archivePath:String, filePath:String)
        Local archive:TReadArchive = New TReadArchive
        archive.SetFormat(EArchiveFormat.Zip)

        ' -----------------------------------
        ' Open archive for reading
        ' -----------------------------------
        ' Attempt to open the ZIP archive for reading
        If archive.Open(archivePath) <> ARCHIVE_OK Then
            Print "Failed to open archive: " + archive.ErrorString()
            archive.Free()
            Return Null
        End If
        
        Local entry:TArchiveEntry = New TArchiveEntry
        
        ' -----------------------------------
        ' Loop through archive entries
        ' -----------------------------------
        ' Search for the requested file in the ZIP archive
        While archive.ReadNextHeader(entry) = ARCHIVE_OK
            Local entryPath:String = entry.Pathname().Replace("\", "/")
            Print "File found in archive: " + entryPath ' Debug
            
            ' -----------------------------------
            ' Check for matching file path
            ' -----------------------------------
            ' Compare paths case-insensitively
            If entryPath.ToLower() = filePath.ToLower() Then
                Local encrypted:Byte[] = New Byte[entry.Size()]

                If archive.Data(encrypted, Size_T(encrypted.Length)) < 0 Then
                    Print "Failed to read data: " + archive.ErrorString()
                    archive.Free()
                    Return Null
                End If

                Local decrypted:Byte[] = DecryptData(encrypted)

                If Not decrypted Then
                    Print "Failed to decrypt file: " + filePath
                    archive.Free()
                    Return Null
                End If

                archive.Free()
                Return String.FromUTF8String(decrypted)
            End If

            archive.DataSkip()

        Wend
        
        archive.Free()
        Print "File not found: " + filePath
        Return Null
    End Method
    
    ' -----------------------------------
    ' Method: ReadFileCustom
    ' -----------------------------------
    ' Internal method to read a file from a custom archive
    Method ReadFileCustom:String(archivePath:String, filePath:String)
        Local stream:TStream = ReadFile(archivePath)

        ' -----------------------------------
        ' Open archive file
        ' -----------------------------------
        ' Open the custom archive file for reading
        If Not stream Then
            Print "Failed to open archive"
            Return Null
        End If
        
        ' Verify header
        Local headerBytes:Byte[] = New Byte[CUSTOM_HEADER.Length]
        stream.ReadBytes(headerBytes, CUSTOM_HEADER.Length)

        ' -----------------------------------
        ' Validate custom header
        ' -----------------------------------
        ' Check if the archive has a valid custom header
        If String.FromUTF8String(headerBytes) <> CUSTOM_HEADER Or stream.ReadByte() <> CUSTOM_VERSION Then
            Print "Invalid format"
            stream.Close()
            Return Null
        End If
        
        Local fileCount:Int = stream.ReadInt()
        
        ' -----------------------------------
        ' Loop through custom archive entries
        ' -----------------------------------
        ' Iterate over entries to find the requested file
        For Local i:Int = 0 Until fileCount
            ' Read path length and path
            Local pathLen:Int = stream.ReadInt()
            Local pathBytes:Byte[] = New Byte[pathLen]
            stream.ReadBytes(pathBytes, pathLen)
            Local path:String = String.FromUTF8String(pathBytes)
            
            ' Read size and data
            Local size:Int = stream.ReadInt()
            Local encrypted:Byte[] = New Byte[size]
            stream.ReadBytes(encrypted, size)
            
            ' -----------------------------------
            ' Check for matching path
            ' -----------------------------------
            ' If the path matches, decrypt and return the content
            If path = filePath Then
                stream.Close()
                Local decrypted:Byte[] = DecryptData(encrypted)

                If Not decrypted Then
                    Print "Failed to decrypt file: " + filePath
                    Return Null
                End If

                Return String.FromUTF8String(decrypted)
            End If

        Next
        
        stream.Close()
        Print "File not found: " + filePath
        Return Null
    End Method
    
    ' -----------------------------------
    ' Method: ExportFile
    ' -----------------------------------
    ' Export a file from an archive to the filesystem
    Method ExportFile:Int(archivePath:String, filePath:String, exportPath:String)
        Local content:String = ReadFile(archivePath, filePath)

        ' -----------------------------------
        ' Check if content was read
        ' -----------------------------------
        ' Verify that file content was successfully retrieved
        If Not content Then
            Return False
        End If
        
        Local stream:TStream = WriteFile(exportPath)

        ' -----------------------------------
        ' Open file for export
        ' -----------------------------------
        ' Attempt to create the output file
        If Not stream Then
            Print "Failed to export file"
            Return False
        End If
        
        Local contentBytes:Byte Ptr = content.ToUTF8String()
        stream.WriteBytes(contentBytes, GetBytePtrLength(contentBytes))
        stream.Close()
        Print "File exported: " + exportPath
        Return True
    End Method
    
    ' -----------------------------------
    ' Method: EncryptData
    ' -----------------------------------
    ' Encrypt data using the secret box encryption
    Method EncryptData:Byte[](data:Byte[])
        Local encrypted:Byte[] = New Byte[data.Length + CRYPTO_SECRETBOX_HEADERBYTES]

        ' -----------------------------------
        ' Perform encryption
        ' -----------------------------------
        ' Encrypt the input data with the key
        If Not TCryptoSecretBox.Encrypt(encrypted, Size_T(encrypted.Length), data, Size_T(data.Length), 0, "archive", key) Then
            Print "Failed to encrypt data"
            Return Null
        End If

        Return encrypted
    End Method
    
    ' -----------------------------------
    ' Method: DecryptData
    ' -----------------------------------
    ' Decrypt data using the secret box decryption
    Method DecryptData:Byte[](encrypted:Byte[])
        Local decrypted:Byte[] = New Byte[encrypted.Length - CRYPTO_SECRETBOX_HEADERBYTES]

        ' -----------------------------------
        ' Perform decryption
        ' -----------------------------------
        ' Decrypt the encrypted data with the key
        If Not TCryptoSecretBox.Decrypt(decrypted, Size_T(decrypted.Length), encrypted, Size_T(encrypted.Length), 0, "archive", key) Then
            Print "Failed to decrypt data"
            Return Null
        End If

        Return decrypted
    End Method
    
    ' -----------------------------------
    ' Method: GetKey
    ' -----------------------------------
    ' Get the encryption key as a string
    Method GetKey:String()
        Return key.ToString()
    End Method
    
    ' -----------------------------------
    ' Method: SetKey
    ' -----------------------------------
    ' Load an existing encryption key from a string
    Method SetKey(keyStr:String)
        key = TCryptoSecretBoxKey.FromString(keyStr)
    End Method
End Type

' -----------------------------------
' Function: FileList
' -----------------------------------
' Recursively list all files in a directory
Function FileList:String[](dir:String, recursive:Int = True)
    Local dirList:String[] = LoadDir(dir)

    ' -----------------------------------
    ' Check directory listing
    ' -----------------------------------
    ' Ensure the directory can be listed
    If Not dirList Then
        Print "Error: Unable to list directory " + dir
        Return New String
 ' Return empty array on failure
    End If
    
    Local list:String[] = New String
    
    ' -----------------------------------
    ' Loop through directory contents
    ' -----------------------------------
    ' Build a list of all files recursively
    For Local file:String = EachIn dirList
        Local fullPath:String = dir + "/" + file
        If FileType(fullPath) = FILETYPE_FILE Then
            list :+ [fullPath]
        ElseIf FileType(fullPath) = FILETYPE_DIR And recursive Then
            list :+ FileList(fullPath, recursive)
        End If
    Next
    Return list
End Function
' -----------------------------------
' Function: LoadByteArray
' -----------------------------------
' Load a file into a byte array
Function LoadByteArray:Byte[](filePath:String)
    Local stream:TStream = ReadFile(filePath)
    ' -----------------------------------
    ' Check file opening
    ' -----------------------------------
    ' Verify the file can be opened for reading
    If Not stream Then 
        Print "Error: Unable to open " + filePath
        Return Null
    End If
    Local data:Byte[] = New Byte[stream.Size()]
    stream.ReadBytes(data, data.Length)
    stream.Close()
    Return data
End Function
' -----------------------------------
' Function: GetBytePtrLength
' -----------------------------------
' Get the length of a Byte Ptr until null terminator
Function GetBytePtrLength:Size_T(bytePtr:Byte Ptr)
    Local length:Size_T = 0
    
    ' -----------------------------------
    ' Loop to find null terminator
    ' -----------------------------------
    ' Count bytes until reaching a null byte
    While bytePtr[length] <> 0
        length:+1
    Wend
    Return length
End Function
' -----------------------------------
' Test Section
' -----------------------------------
' Test the archive manager functionality
Local manager:TArchiveManager = New TArchiveManager("monMotDePasse", False) ' False = ZIP, True = custom
Local key:String = manager.GetKey()
Print "Generated key: " + key
' Archiver un répertoire
manager.ArchiveDirectory("directory", "archive.zip")
' Lire un fichier
Local content:String = manager.ReadFile("archive.zip", "directory/child/Salutation.txt")
If content Then
    Print "Content: " + content
End If
' Exporter un fichier
manager.ExportFile("archive.zip", "directory/child/Salutation.txt", "exported.txt")
' Tester avec une nouvelle instance et la même clé
Local manager2:TArchiveManager = New TArchiveManager("")
manager2.SetKey(key)
content = manager2.ReadFile("archive.zip", "directory/child/Salutation.txt")
If content Then
    Print "Content (new instance): " + content
End If

Filax

#5
April 11, 2025, 07:21:46 Last Edit: April 11, 2025, 23:03:18 by Filax
A better version with threads! :)

Code: BASIC
' ------------------------------------------------
' Name : Data Package System V0.6.1 - Debugged
' Date : (C)2025
' Site : https://github.com/BlackCreepyCat
' Description : Compression -> Encryption -> Storage (No key storage, thread debug)
' ------------------------------------------------
SuperStrict

Import BRL.IO
Import Crypto.Crypto
Import Archive.Zip
Import archive.Core
Import archive.ZLib
Import BRL.Base64
Import BRL.Threads

Type TArchiveManager
    Field archive:TWriteArchive
    Field readArchive:TReadArchive
    Field fileList:TList
    Field secretKey:TCryptoSecretBoxKey
    Field password:String
    Field context:String = "archive"
    Const CRYPTO_SECRETBOX_HEADERBYTES:Int = 36
    Field rootDir:String
    Field archiveMutex:TMutex
    Field compressThreads:TThread[4]
    Field threadErrors:Int
    Field filesCompressed:Int

    Method New(archivePath:String, password:String, key:TCryptoSecretBoxKey)
        Self.password = password
        Self.secretKey = key
        If Not Self.secretKey Then Throw "Error: No encryption key provided"
        Self.archive = New TWriteArchive
        Print "Archive: " + Self.archive.ToString()
        
        If FileType(archivePath) = FILETYPE_FILE Then
            DeleteFile(archivePath)
            Print "Existing file deleted: " + archivePath
        EndIf
        
        Self.archive.SetFormat(EArchiveFormat.ZIP)
        Print "Attempting to open archive: " + archivePath
        
        If Self.archive.Open(archivePath) <> ARCHIVE_OK Then
            Throw "Error opening archive for writing: " + Self.archive.ErrorString()
        EndIf
        
        Print "Archive opened successfully"
        Self.archiveMutex = CreateMutex()
        Self.fileList = New TList
        Self.threadErrors = 0
        Self.filesCompressed = 0
    End Method

    Method OpenForRead(archivePath:String, password:String, key:TCryptoSecretBoxKey)
        Self.password = password
        Self.secretKey = key
        If Not Self.secretKey Then Throw "Error: No encryption key provided"
        Self.readArchive = New TReadArchive
        Self.readArchive.SetFormat(EArchiveFormat.ZIP)
        Print "Attempting to open existing archive for reading: " + archivePath
        
        If FileType(archivePath) <> FILETYPE_FILE Then
            Throw "Archive file does not exist: " + archivePath
        EndIf
        
        If Self.readArchive.Open(archivePath) <> ARCHIVE_OK Then
            Throw "Error opening existing archive for reading: " + Self.readArchive.ErrorString()
        EndIf
        
        Print "Existing archive opened for reading successfully"
    End Method

    Method ScanDirectory(dirPath:String)
        Self.rootDir = dirPath
        Local files:TList = New TList
        
        Print "Starting directory scan: " + dirPath
        ScanSubDirectory(dirPath, files)
        
        Self.fileList = files
        Print "Files scanned: " + Self.fileList.Count()
        
        For Local file:String = EachIn fileList
            Print " - " + file
        Next
    End Method

    Method ScanSubDirectory(dirPath:String, files:TList)
        Local dir:Byte Ptr = ReadDir(dirPath)
        
        If Not dir Then
            Print "Unable to open directory: " + dirPath
            Return
        EndIf
        
        Local entry:String = NextFile(dir)
        
        While entry
            If entry <> "." And entry <> ".." Then
                Local fullPath:String = dirPath + "/" + entry
                Local relativePath:String = fullPath[rootDir.Length + 1..]
                
                If FileType(fullPath) = FILETYPE_FILE Then
                    files.AddLast(relativePath)
                    Print "File found: " + relativePath
                ElseIf FileType(fullPath) = FILETYPE_DIR Then
                    Print "Subdirectory found: " + relativePath
                    ScanSubDirectory(fullPath, files)
                EndIf
            EndIf
            entry = NextFile(dir)
        Wend
        
        CloseDir(dir)
    End Method

    Function CompressFileThread:Object(data:Object)
        Local params:TCompressParams = TCompressParams(data)
        Local manager:TArchiveManager = params.manager
        Local filePath:String = params.filePath
        Local startTime:Double = MilliSecs()
        
        Print "Thread started for: " + filePath
        
        Local entry:TArchiveEntry = New TArchiveEntry
        entry.SetPathname(filePath)
        entry.SetFileType(EArchiveFileType.File)

        Local fullPath:String = manager.rootDir + "/" + filePath
        Print "Loading file: " + fullPath
        Local content:Byte[] = manager.LoadByteArray(fullPath)
        
        If Not content Then
            Print "ERROR: Failed to load file: " + fullPath
            LockMutex(manager.archiveMutex)
            manager.threadErrors :+ 1
            UnlockMutex(manager.archiveMutex)
            Return Null
        EndIf
        
        Local originalSize:ULongInt = ULongInt(content.Length)
        
        Print "Prefixing content for: " + fullPath
        Local prefixedContent:Byte[] = New Byte[content.Length + 9]
        For Local i:Int = 0 To 7
            prefixedContent[i] = (originalSize Shr (i * 8)) & $FF
        Next
        MemCopy(VarPtr prefixedContent[9], VarPtr content, Size_T(content.Length))
        
        Print "Compressing content for: " + fullPath
        Local maxCompressedSize:ULongInt = ULongInt(prefixedContent.Length) + (ULongInt(prefixedContent.Length) / 10) + 12
        Local compressedContent:Byte[] = New Byte[maxCompressedSize]
        Local compressedSize:ULongInt = maxCompressedSize
        Local result:Int = compress2(Varptr compressedContent, compressedSize, Varptr prefixedContent, ULongInt(prefixedContent.Length), 6)
        Local finalContent:Byte[]
        
        If result = 0 And compressedSize < ULongInt(prefixedContent.Length) Then
            compressedContent = compressedContent[..compressedSize]
            finalContent = New Byte[compressedSize + 1]
            finalContent
 = 1 ' Flag: compressed
            MemCopy(VarPtr finalContent[1], VarPtr compressedContent, Size_T(compressedSize))
            Print "Compression succeeded for: " + fullPath
        Else
            finalContent = New Byte[prefixedContent.Length + 1]
            finalContent = 0 ' Flag: not compressed
            MemCopy(VarPtr finalContent[1], VarPtr prefixedContent, Size_T(prefixedContent.Length))
            Print "Compression skipped for: " + fullPath + " (size unchanged or increased)"
        EndIf
        
        Print "Encrypting content for: " + fullPath
        Local encryptedContent:Byte[]
        Try
            encryptedContent = manager.EncryptData(finalContent)
            If Not encryptedContent Then
                Print "ERROR: Encryption returned null for: " + fullPath
                LockMutex(manager.archiveMutex)
                manager.threadErrors :+ 1
                UnlockMutex(manager.archiveMutex)
                Return Null
            EndIf
        Catch e:String
            Print "ERROR: Encryption failed for: " + fullPath + " (" + e + ")"
            LockMutex(manager.archiveMutex)
            manager.threadErrors :+ 1
            UnlockMutex(manager.archiveMutex)
            Return Null
        EndTry
        
        Local encryptedSize:ULongInt = ULongInt(encryptedContent.Length)
        
        Print "Writing header for: " + fullPath
        entry.SetSize(Long(encryptedSize))
        entry.SetModifiedTime(FileTime(fullPath))
        LockMutex(manager.archiveMutex)
        
        If manager.archive.Header(entry) <> ARCHIVE_OK Then
            Print "ERROR: Header failed for: " + fullPath + " : " + manager.archive.ErrorString()
            manager.threadErrors :+ 1
            UnlockMutex(manager.archiveMutex)
            Return Null
        EndIf
        
        Print "Writing data for: " + fullPath
        If manager.archive.Data(encryptedContent, Size_T(encryptedContent.Length)) < 0 Then
            Print "ERROR: Data write failed for: " + fullPath + " : " + manager.archive.ErrorString()
            manager.threadErrors :+ 1
            UnlockMutex(manager.archiveMutex)
            Return Null
        EndIf
        
        Print "Finishing entry for: " + fullPath
        If manager.archive.FinishEntry() <> ARCHIVE_OK Then
            Print "ERROR: Entry finalization failed for: " + fullPath + " : " + manager.archive.ErrorString()
            manager.threadErrors :+ 1
            UnlockMutex(manager.archiveMutex)
            Return Null
        EndIf
        
        manager.filesCompressed :+ 1
        Local progress:Float = (Float(manager.filesCompressed) / Float(manager.fileList.Count())) * 100
        UnlockMutex(manager.archiveMutex)
        
        Local endTime:Double = MilliSecs()
        Local totalTime:Double = (endTime - startTime) / 1000.0
        Print filePath + " / Original size: " + originalSize + " / Final size: " + encryptedSize + " / Total time: " + totalTime + "s / Progress: " + progress + "%"
        Print "Thread completed for: " + filePath
        Return Null
    End Function
    Method CreateArchive()
        If Not fileList Or fileList.Count() = 0 Then
            Throw "No files to archive"
        EndIf
        Local threadCount:Int = Min(4, fileList.Count())
        Local filesProcessed:Int = 0
        
        Print "Starting archive creation with " + fileList.Count() + " files"
        
        For Local i:Int = 0 Until Min(threadCount, fileList.Count())
            Local file:String = String(fileList.ValueAtIndex(i))
            Local params:TCompressParams = New TCompressParams
            params.manager = Self
            params.filePath = file
            compressThreads[i] = CreateThread(CompressFileThread, params)
            filesProcessed :+ 1
            Print "Thread launched for: " + file
        Next
        
        While filesProcessed < fileList.Count()
            Print "Checking threads: " + filesProcessed + " / " + fileList.Count()
            For Local j:Int = 0 Until threadCount
                If compressThreads[j] Then
                    If Not ThreadRunning(compressThreads[j]) Then
                        WaitThread(compressThreads[j])
                        Print "Thread " + j + " finished for: " + String(fileList.ValueAtIndex(filesProcessed - 1))
                        compressThreads[j] = Null
                        
                        Local file:String = String(fileList.ValueAtIndex(filesProcessed))
                        Local params:TCompressParams = New TCompressParams
                        params.manager = Self
                        params.filePath = file
                        compressThreads[j] = CreateThread(CompressFileThread, params)
                        filesProcessed :+ 1
                        Print "Thread relaunched for: " + file
                        Exit
                    Else
                        Print "Thread " + j + " still running"
                    EndIf
                EndIf
            Next
            Delay(100) ' Petit délai pour éviter une boucle trop agressive
        Wend
        
        For Local i:Int = 0 Until threadCount
            If compressThreads[i] Then
                WaitThread(compressThreads[i])
                Print "Thread " + i + " finished"
            EndIf
        Next
        
        If threadErrors > 0 Then
            Print "ERROR: Errors detected in " + threadErrors + " threads"
        EndIf
        
        If Self.archive.Close() <> ARCHIVE_OK Then
            Throw "Error closing archive: " + Self.archive.ErrorString()
        EndIf
        
        Print "Archive created successfully"
    End Method
    Method ExtractSingleFile(archivePath:String, filePathInArchive:String, destDir:String, key:TCryptoSecretBoxKey, keepHierarchy:Int = True)
        Self.secretKey = key
        If Not Self.secretKey Then Throw "Error: No encryption key provided"
        Self.readArchive = New TReadArchive
        Self.readArchive.SetFormat(EArchiveFormat.ZIP)
        
        If Self.readArchive.Open(archivePath) <> ARCHIVE_OK Then
            Throw "Error opening archive for reading: " + Self.readArchive.ErrorString()
        EndIf
        
        Local entry:TArchiveEntry = New TArchiveEntry
        Local found:Int = False
        Local adjustedPath:String = filePathInArchive
        
        While Self.readArchive.ReadNextHeader(entry) = ARCHIVE_OK
            Local relativePath:String = entry.Pathname()
            Print "Entry in archive: " + relativePath
            
            If relativePath = adjustedPath Then
                found = True
                Print "File found in archive: " + relativePath
                
                Local destPath:String
                If keepHierarchy Then
                    destPath = destDir + "/" + relativePath
                Else
                    Local fileName:String = relativePath[relativePath.FindLast("/") + 1..]
                    destPath = destDir + "/" + fileName
                EndIf
                
                CreateDir(ExtractDir(destPath), True)
                Local outStream:TStream = WriteFile(destPath)
                
                If Not outStream Then
                    Self.readArchive.Free()
                    Throw "Failed to create destination file: " + destPath
                EndIf
                
                Local encryptedContent:Byte[] = New Byte[entry.Size()]
                If Self.readArchive.Data(encryptedContent, Size_T(encryptedContent.Length)) < 0 Then
                    CloseStream(outStream)
                    Self.readArchive.Free()
                    Throw "Error reading data from archive: " + Self.readArchive.ErrorString()
                EndIf
                
                Print "Encrypted data size: " + encryptedContent.Length
                
                Local decryptedContent:Byte[]
                Try
                    decryptedContent = DecryptData(encryptedContent)
                Catch e:String
                    CloseStream(outStream)
                    Self.readArchive.Free()
                    Throw "Failed to decrypt data for: " + relativePath + " (" + e + ")"
                EndTry
                
                If Not decryptedContent Then
                    CloseStream(outStream)
                    Self.readArchive.Free()
                    Throw "Failed to decrypt data for: " + relativePath
                EndIf
                
                Local isCompressed:Byte = decryptedContent
                Local dataToDecompress:Byte[] = decryptedContent[1..]
                Local finalContent:Byte[]
                         Local originalSize:ULongInt = 0
						          
                If isCompressed Then
                    Local estimatedDecompressedSize:ULongInt = ULongInt(dataToDecompress.Length) * 10
                    Local decompressedContent:Byte[] = New Byte[estimatedDecompressedSize]
                    Local decompressedSize:ULongInt = estimatedDecompressedSize
                    Local result:Int = uncompress(Varptr decompressedContent, decompressedSize, Varptr dataToDecompress, ULongInt(dataToDecompress.Length))
   
                    If result <> 0 Then
                        CloseStream(outStream)
                        Self.readArchive.Free()
                        Throw "Failed to decompress data for: " + relativePath + " (Error code: " + result + ")"
                    EndIf
                    
                    Local originalSize:ULongInt = 0
                    For Local i:Int = 0 To 7
                        originalSize :| (ULongInt(decompressedContent[i]) Shl (i * 8))
                    Next
                    finalContent = decompressedContent[8..8 + originalSize]
                Else
                   
                    For Local i:Int = 0 To 7
                        originalSize :| (ULongInt(dataToDecompress[i]) Shl (i * 8))
                    Next
                    finalContent = dataToDecompress[8..8 + originalSize]
                EndIf
                
                outStream.WriteBytes(finalContent, Long(finalContent.Length))
                CloseStream(outStream)
                Print "File successfully extracted to: " + destPath + " (Original size: " + originalSize + ")"
                Exit
            Else
                Local tempBuffer:Byte[] = New Byte[entry.Size()]
                If Self.readArchive.Data(tempBuffer, Size_T(tempBuffer.Length)) < 0 Then
                    Self.readArchive.Free()
                    Throw "Error skipping unwanted data for: " + relativePath
                EndIf
            EndIf
        Wend
        
        Self.readArchive.Free()
        
        If Not found Then
            Throw "File not found in archive: " + filePathInArchive
        EndIf
    End Method
    Method EncryptData:Byte[](data:Byte[])
        Local msgId:ULong = 1234567890
        Local encrypted:Byte[data.Length + CRYPTO_SECRETBOX_HEADERBYTES]
        
        If Not TCryptoSecretBox.Encrypt(encrypted, Size_T(data.Length + CRYPTO_SECRETBOX_HEADERBYTES), data, Size_T(data.Length), msgId, context, secretKey) Then
            Throw "Failed to encrypt data"
        EndIf
        
        Return encrypted
    End Method
    Method DecryptData:Byte[](encrypted:Byte[])
        Local msgId:ULong = 1234567890
        Local decrypted:Byte[encrypted.Length - CRYPTO_SECRETBOX_HEADERBYTES]
        
        If Not TCryptoSecretBox.Decrypt(decrypted, Size_T(decrypted.Length), encrypted, Size_T(encrypted.Length), msgId, context, secretKey) Then
            Throw "Failed to decrypt data"
        EndIf
        
        Return decrypted
    End Method
    Function LoadByteArray:Byte[](filePath:String)
        Local stream:TStream = ReadFile(filePath)
        
        If Not stream Then
            Print "Error: Unable to read file: " + filePath
            Return Null
        EndIf
        
        Local data:Byte[] = New Byte[stream.Size()]
        stream.ReadBytes(data, data.Length)
        stream.Close()
        Return data
    End Function
    Method Free()
        If archive Then
            archive.Free()
        EndIf
        
        If readArchive Then
            readArchive.Free()
        EndIf
        
        If archiveMutex Then
            CloseMutex(archiveMutex)
        EndIf
    End Method
End Type
Type TCompressParams
    Field manager:TArchiveManager
    Field filePath:String
End Type
Function Main()
    Try
        Local key:TCryptoSecretBoxKey = TCryptoSecretBox.KeyGen()
        
        Local archiver:TArchiveManager = New TArchiveManager("test.zip", "monMotDePasse", key)
        archiver.ScanDirectory("D:\Downloads\bah.mod-master")
        archiver.CreateArchive()
        
        Local existingArchiver:TArchiveManager = New TArchiveManager
        existingArchiver.OpenForRead("test.zip", "monMotDePasse", key)
        existingArchiver.ExtractSingleFile("test.zip", "readme_archived.md", CurrentDir(), key, False)
        
        archiver.Free()
        existingArchiver.Free()
    
    Catch e:String
        Print "Error: " + e
    End Try
End Function
Main()

PixelOutlaw

#6
April 12, 2025, 07:19:56
Interesting using the "/" separator...
Is Windows allowing that in file paths rather than "\" these days?
I guess that fixes a long standing ambiguity between OS paths if so.
Ubuntu MATE 20.04: i5-3570K CPU @ 3.40GHz, 8GB RAM, GeForce GTX 1060 3GB

One DEFUN to rule them all, One DEFUN to find them, One DEFUN to RETURN them all, and in the darkness MULTIPLE-VALUE-BIND them.

dawlane

#7
April 12, 2025, 07:54:34 Last Edit: April 12, 2025, 08:03:54 by dawlane
Quote from: PixelOutlaw on April 12, 2025, 07:19:56Interesting using the "/" separator...
Is Windows allowing that in file paths rather than "\" these days?
I guess that fixes a long standing ambiguity between OS paths if so.
Both the solidus "'/" and reverse solidus "\" can be used as file path separators ever since the days of DOS 2.0, but a solidus "/" was never meant to be a file path separator in that operating system. The solidus was meant to be an application option specifier. So care has to be taken when passing a command line to the MS DOS/MS Windows command shell when doing cross platform programming. All paths should be encased in quotation marks to prevent the shell interpreter from reading parts of the path as a command line option.

NOTE: Solidus is only one of the names that the symbol is know by.
Print
Go Up Pages1
User actions