[bb] LZMA Streams by Otus [ 1+ years ago ]

Started by BlitzBot, June 29, 2017, 00:28:38

Previous topic - Next topic

BlitzBot

Title : LZMA Streams
Author : Otus
Posted : 1+ years ago

Description : TLzmaStream wraps another stream (eg. a file stream) and handles compression automatically. This makes reading and writing compressed files as easy as using the "lzma::" prefix when opening files.

Uses <a href="http://www.blitzmax.com/codearcs/codearcs.php?code=2377" target="_blank">LZMA Compression</a>, so you need to have that. Module versions of both are here: <a href="http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/" target="_blank">http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/</a>

Note: The wrapped stream needs to support seeking.

Note2: If the compressed data would take more space, uncompressed data is written instead. Worst case +4bytes for header.


Code :
Code (blitzbasic) Select
SuperStrict

Import BRL.BankStream
Import BRL.Stream

Import "lzma.bmx"

Rem
bbdoc: LZMA stream wrapper type
about:
#TLzmaStream wraps a raw stream and allows access to uncompressed data.

When writing, the data is compressed and the compressed daa written to the wrapped stream.
If compression expands the data, uncompressed data is written instead.

Changes in the raw stream don't automatically appear in a TLzmaStream
- #ReadSync updates to the current raw stream, but any changes are lost.

Similarly, changes written to a TLzmaStream are only written to the raw stream
on a Flush/FlushStream call or when the stream is closed.

Note: You may lose data if you fail to close/flush the stream before program ends.
Do not rely on the automatic Delete->Close call!
End Rem
Type TLzmaStream Extends TStreamWrapper

Field _basestream:TStream

Field _level:Int = 5

Rem
bbdoc: Closes the stream, writing any changes
End Rem
Method Close()
Flush()
_basestream.Close()
_stream.Close()
End Method

Rem
bbdoc: Updates to current raw stream data
End Rem
Method ReadSync()
'Empty stream?
If _basestream.Size()=0
_stream = CreateBankStream(Null)
Return
End If

'Copy stream contents to a bank
Local b:TBank = CreateBank(_basestream.Size())
CopyStream _basestream, CreateBankStream(b)

'Set up bank for raw access
Local buf:Byte Ptr = b.Lock()
Local size:Int = b.Size()-4


'Is this uncompressed data?
Local usize:Int = Int Ptr(buf)[0] + 1
If usize<=1
Local u:TBank = CreateBank(-usize)
Local ubuf:Byte Ptr = u.Lock()
MemCopy ubuf, buf+4, -usize
u.Unlock()
_stream = CreateBankStream(u)
Return
End If


'Create a bank for uncompressed data
Local u:TBank = CreateBank(usize)
Local ubuf:Byte Ptr = u.Lock()

LzmaUncompress ubuf, usize, buf+4, size

'Not valid LZMA?
If usize <> u.Size()-1 Then Return

u.Unlock()
u.Resize(usize)

_stream = CreateBankStream(u)
End Method

Rem
bbdoc: Flushes current data to the raw stream
End Rem
Method Flush()
'Set up bank for raw access
Local b:TBank = TBankStream(_stream)._bank
Local bsize:Int = b.Size()
Local buf:Byte Ptr = b.Lock()

'Create bank for compressed data
Local csize:Int = bsize + 1024
Local c:TBank = CreateBank(csize)
Local cbuf:Byte Ptr = c.Lock()

LzmaCompress2 cbuf, csize, buf, bsize, _level

_basestream.Seek 0

'Does it fit?
If csize<b.Size()
_basestream.WriteInt b.Size()
_basestream.WriteBytes cbuf, csize
Else
'Write uncompressed
_basestream.WriteInt -b.Size()
_basestream.WriteBytes buf, b.Size()
End If

b.Unlock()
End Method

Function Create:TLzmaStream( stream:TStream )
'Stream must be seekable
If stream=Null Or stream.Seek(0)=-1 Then Return Null

Local l:TLzmaStream = New TLzmaStream
l._basestream = stream
l.ReadSync()

If Not l._stream Then Return Null

Return l
End Function

End Type

Rem
bbdoc: Opens #url as TLzmaStream
about:
An alternative to using OpenStream("lzma::-blah").
End Rem
Function CreateLzmaStream:TLzmaStream( url:Object )
Return TLzmaStream.Create( OpenStream(url) )
End Function

New TLzmaStreamFactory

Type TLzmaStreamFactory Extends TStreamFactory

Method CreateStream:TStream( url:Object,proto$,path$,readable%,writeable% )
If proto<>"lzma" Then Return Null
Local stream:TStream = OpenStream(path)
Assert stream<>Null
Return TLzmaStream.Create( stream )
End Method

End Type


Comments :


Otus(Posted 1+ years ago)

 A simple sample:
SuperStrict

Framework BRL.StandardIO

Import BRL.FileSystem

Import "lzmastream.bmx"

'Write a compressed file

Local out:TStream = WriteStream("lzma::test.txt")

For Local i% = 1 To 10
out.WriteLine "Hello World"
out.WriteLine "This is a test."
Next

out.Close()

'Read the compressed file

Local in:TStream = ReadStream("lzma::test.txt")

While Not in.Eof()
Print in.ReadLine()
Wend



Otus(Posted 1+ years ago)

 Language should have been .bmx of course. Is there any way to change that, I wonder...


markcw(Posted 1+ years ago)

 Mods can do that if they notice. You could add [bmax] or [bmx] to the short description.


plash(Posted 1+ years ago)

 You should stick this in a module, you seem to already have pretty much everything documented and set for a module.


xlsior(Posted 1+ years ago)

 Oooooh... Very useful!Thanks for sharing.


Otus(Posted 1+ years ago)

 <div class="quote"> You should stick this in a module, you seem to already have pretty much everything documented and set for a module.  </div>As I wrote in the post, the module version is here: <a href="http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/" target="_blank">http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/</a>


plash(Posted 1+ years ago)

 Doh. Thanks. [/i]