November 24, 2020, 06:10:24 AM

Author Topic: [bb] LZMA Streams by Otus [ 1+ years ago ]  (Read 762 times)

Offline BlitzBot

  • Jr. Member
  • **
  • Posts: 1
[bb] LZMA Streams by Otus [ 1+ years ago ]
« on: June 29, 2017, 12:28:38 AM »
Title : LZMA Streams
Author : Otus
Posted : 1+ years ago

Description : TLzmaStream wraps another stream (eg. a file stream) and handles compression automatically. This makes reading and writing compressed files as easy as using the "lzma::" prefix when opening files.

Uses <a href="http://www.blitzmax.com/codearcs/codearcs.php?code=2377" target="_blank">LZMA Compression[/url], so you need to have that. Module versions of both are here: <a href="http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/" target="_blank">http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/[/url]

Note: The wrapped stream needs to support seeking.

Note2: If the compressed data would take more space, uncompressed data is written instead. Worst case +4bytes for header.


Code :
Code: BlitzBasic
  1. SuperStrict
  2.  
  3. Import BRL.BankStream
  4. Import BRL.Stream
  5.  
  6. Import "lzma.bmx"
  7.  
  8. Rem
  9. bbdoc: LZMA stream wrapper type
  10. about:
  11. #TLzmaStream wraps a raw stream and allows access to uncompressed data.
  12.  
  13. When writing, the data is compressed and the compressed daa written to the wrapped stream.
  14. If compression expands the data, uncompressed data is written instead.
  15.  
  16. Changes in the raw stream don't automatically appear in a TLzmaStream
  17. - #ReadSync updates to the current raw stream, but any changes are lost.
  18.  
  19. Similarly, changes written to a TLzmaStream are only written to the raw stream
  20. on a Flush/FlushStream call or when the stream is closed.
  21.  
  22. Note: You may lose data if you fail to close/flush the stream before program ends.
  23. Do not rely on the automatic Delete->Close call!
  24. End Rem
  25. Type TLzmaStream Extends TStreamWrapper
  26.        
  27.         Field _basestream:TStream
  28.        
  29.         Field _level:Int = 5
  30.        
  31. Rem
  32. bbdoc: Closes the stream, writing any changes
  33. End Rem
  34.         Method Close()
  35.                 Flush()
  36.                 _basestream.Close()
  37.                 _stream.Close()
  38.         End Method
  39.        
  40. Rem
  41. bbdoc: Updates to current raw stream data
  42. End Rem
  43.         Method ReadSync()
  44.                 'Empty stream?
  45.                 If _basestream.Size()=0
  46.                         _stream = CreateBankStream(Null)
  47.                         Return
  48.                 End If
  49.                
  50.                 'Copy stream contents to a bank
  51.                 Local b:TBank = CreateBank(_basestream.Size())
  52.                 CopyStream _basestream, CreateBankStream(b)
  53.                
  54.                 'Set up bank for raw access
  55.                 Local buf:Byte Ptr = b.Lock()
  56.                 Local size:Int = b.Size()-4
  57.                
  58.                
  59.                 'Is this uncompressed data?
  60.                 Local usize:Int = Int Ptr(buf)[0] + 1
  61.                 If usize<=1
  62.                         Local u:TBank = CreateBank(-usize)
  63.                         Local ubuf:Byte Ptr = u.Lock()
  64.                         MemCopy ubuf, buf+4, -usize
  65.                         u.Unlock()
  66.                         _stream = CreateBankStream(u)
  67.                         Return
  68.                 End If
  69.                
  70.                
  71.                 'Create a bank for uncompressed data
  72.                 Local u:TBank = CreateBank(usize)
  73.                 Local ubuf:Byte Ptr = u.Lock()
  74.                
  75.                 LzmaUncompress ubuf, usize, buf+4, size
  76.                
  77.                 'Not valid LZMA?
  78.                 If usize <> u.Size()-1 Then Return
  79.                
  80.                 u.Unlock()
  81.                 u.Resize(usize)
  82.                
  83.                 _stream = CreateBankStream(u)
  84.         End Method
  85.        
  86. Rem
  87. bbdoc: Flushes current data to the raw stream
  88. End Rem
  89.         Method Flush()
  90.                 'Set up bank for raw access
  91.                 Local b:TBank = TBankStream(_stream)._bank
  92.                 Local bsize:Int = b.Size()
  93.                 Local buf:Byte Ptr = b.Lock()
  94.                
  95.                 'Create bank for compressed data
  96.                 Local csize:Int = bsize + 1024
  97.                 Local c:TBank = CreateBank(csize)
  98.                 Local cbuf:Byte Ptr = c.Lock()
  99.                
  100.                 LzmaCompress2 cbuf, csize, buf, bsize, _level
  101.                
  102.                 _basestream.Seek 0
  103.                
  104.                 'Does it fit?
  105.                 If csize<b.Size()
  106.                         _basestream.WriteInt b.Size()
  107.                         _basestream.WriteBytes cbuf, csize
  108.                 Else
  109.                         'Write uncompressed
  110.                         _basestream.WriteInt -b.Size()
  111.                         _basestream.WriteBytes buf, b.Size()
  112.                 End If
  113.                
  114.                 b.Unlock()
  115.         End Method
  116.        
  117.         Function Create:TLzmaStream( stream:TStream )
  118.                 'Stream must be seekable
  119.                 If stream=Null Or stream.Seek(0)=-1 Then Return Null
  120.                
  121.                 Local l:TLzmaStream = New TLzmaStream
  122.                 l._basestream = stream
  123.                 l.ReadSync()
  124.                
  125.                 If Not l._stream Then Return Null
  126.                
  127.                 Return l
  128.         End Function
  129.        
  130. End Type
  131.  
  132. Rem
  133. bbdoc: Opens #url as TLzmaStream
  134. about:
  135. An alternative to using OpenStream("lzma::-blah").
  136. End Rem
  137. Function CreateLzmaStream:TLzmaStream( url:Object )
  138.         Return TLzmaStream.Create( OpenStream(url) )
  139. End Function
  140.  
  141. New TLzmaStreamFactory
  142.  
  143. Type TLzmaStreamFactory Extends TStreamFactory
  144.        
  145.         Method CreateStream:TStream( url:Object,proto$,path$,readable%,writeable% )
  146.                 If proto<>"lzma" Then Return Null
  147.                 Local stream:TStream = OpenStream(path)
  148.                 Assert stream<>Null
  149.                 Return TLzmaStream.Create( stream )
  150.         End Method
  151.        
  152. End Type


Comments :


Otus(Posted 1+ years ago)

 A simple sample:
Code: [Select]
SuperStrict

Framework BRL.StandardIO

Import BRL.FileSystem

Import "lzmastream.bmx"

'Write a compressed file

Local out:TStream = WriteStream("lzma::test.txt")

For Local i% = 1 To 10
out.WriteLine "Hello World"
out.WriteLine "This is a test."
Next

out.Close()

'Read the compressed file

Local in:TStream = ReadStream("lzma::test.txt")

While Not in.Eof()
Print in.ReadLine()
Wend



Otus(Posted 1+ years ago)

 Language should have been .bmx of course. Is there any way to change that, I wonder...


markcw(Posted 1+ years ago)

 Mods can do that if they notice. You could add [bmax] or [bmx] to the short description.


plash(Posted 1+ years ago)

 You should stick this in a module, you seem to already have pretty much everything documented and set for a module.


xlsior(Posted 1+ years ago)

 Oooooh... Very useful!Thanks for sharing.


Otus(Posted 1+ years ago)

 <div class="quote"> You should stick this in a module, you seem to already have pretty much everything documented and set for a module.  </div>As I wrote in the post, the module version is here: <a href="http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/" target="_blank">http://jan.varho.org/blog/programming/blitz/lzma-streams-for-blitzmax/[/url]


plash(Posted 1+ years ago)

 Doh. Thanks. [/i]

 

SimplePortal 2.3.6 © 2008-2014, SimplePortal