December 03, 2020, 08:28:15 PM

Author Topic: [bmx] Simple Parse by N [ 1+ years ago ]  (Read 542 times)

Offline BlitzBot

  • Jr. Member
  • **
  • Posts: 1
[bmx] Simple Parse by N [ 1+ years ago ]
« on: June 29, 2017, 12:28:40 AM »
Title : Simple Parse
Author : N
Posted : 1+ years ago

Description : The general idea behind SParse was to rip off the Quake 3 shader system.  That is, stuff that looked like this:
Code: [Select]
textures/base/floor_01_fx
{
    somestuff
    { // a pass
        texmod somesuch 0 5
        map textures/base/floor_01_fx.tga
    }
    { // another pass
        you get the idea
    }
}


Well, I did that, albeit with some changes in syntax and then making it a generic system for reading text structured similar to that.  SParse looks like this:

Code: [Select]
#comment
# a root-level attribute
attribute value
valuelessAttr!

{ #nameless node
    randomName value
    anotherAttr some more stuff for this
 across multiple lines
 using escape values like \n
    someChildNode
    {
        anotherNamelessAttr!
        value attribute's name is value
    }
}

namedNode {
    attribute value
    you get the idea
    # ^ this would be an attribute called 'you' with a value
    # of 'get the idea'
}


So, goal accomplished, I wrote a nice little way to store simple data in text form.  Everything is documented to my knowledge, and I don't much care beyond that.  I use this regularly for my own work, I've ported it to various other languages, and it's been very useful to me, so I'm sharing it.  Enjoy. [/i]

Code :
Code: BlitzMax
  1. SuperStrict
  2.  
  3. Import "collections.bmx"
  4.  
  5. Private
  6. Global buf@[8192] ' Buffer of data to be parsed
  7. Global shrts:Short[128] ' Buffer for what used to be nam$ in IPNode.ReadNodes
  8.  
  9. Global gIPNodeDataType:IPNodeType = New IPNodeType
  10. Global gIPDocDataType:IPDocType = New IPDocType
  11.  
  12. Type IPDocType Extends TDataType
  13.     Method Tag$( )
  14.         Return "Parse.IPDoc"
  15.     End Method
  16.    
  17.     Method WriteObject( obj:Object, stream:TStream )
  18.         IPDoc(obj).Write(stream)
  19.     End Method
  20.  
  21.     Method ReadObject:Object( stream:TStream )
  22.         Return IPDoc.Load(stream)
  23.     End Method
  24. End Type
  25.  
  26. Type IPNodeType Extends TDataType
  27.     Method Tag$( )
  28.         Return "Parse.IPNode"
  29.     End Method
  30.  
  31.     Method WriteObject( obj:Object, stream:TStream )
  32.         Local node:IPNode = IPNode( obj )
  33.         If Not node Then Return
  34.         node.Write( stream )
  35.     End Method
  36.  
  37.     Method ReadObject:Object( stream:TStream )
  38.         Local node:IPNode = New IPNode
  39.         node.Read( stream, -1 )
  40.         Return node
  41.     End Method
  42. End Type
  43.  
  44. Public
  45.  
  46. Rem
  47. bbdoc: Sets the size of the buffer that's parsed.  When Parse is parsing a document, it reads a maximum amount of @size bytes to parse and parses them.  This repeats until the document reaches its maximum position (typically either EOF or until a specified size limit is reached).
  48. notes: The default @size is 8192 (8kb -- shouldn't be too much for most cases)
  49. EndRem
  50. Function SetParseCacheSize( size%=8192 )
  51.     buf = New Byte[size]
  52. End Function
  53.  
  54. Rem
  55. bbdoc: Managed interface for a collection of IPNodes.
  56. EndRem
  57. Type IPDoc Extends TData
  58.     Rem
  59.     bbdoc: The root of the IPDoc
  60.     EndRem
  61.     Field Root:IPNode
  62.  
  63.     Method Delete()
  64.         root.Dispose()
  65.         root = Null
  66.     End Method
  67.  
  68.     Method New()
  69.         root = New IPNode
  70.     End Method
  71.  
  72.     Rem
  73.     bbdoc: Load an IPDoc from a @source.
  74.     EndRem
  75.     Function Load:IPDoc( source:Object )
  76.         If Not source Then
  77.             Throw "Null URL passed to LoadParseDoc"
  78.         EndIf
  79.  
  80.         Local stream:TStream = TStream(source)
  81.         If stream = Null Then stream = ReadStream(source)
  82.  
  83.         If Not stream Then
  84.             Throw "Failed to read stream ["+source.ToString( )+"]"
  85.         EndIf
  86.  
  87.         Local doc:IPDoc = New IPDoc
  88.  
  89.         doc.root.Read( stream )
  90.  
  91.         Return doc
  92.     End Function
  93.  
  94.     Rem
  95.     bbdoc: Write an IPDoc to a @stream.
  96.     EndRem
  97.     Method Write:Int( stream:TStream )
  98.         If root = Null Then
  99.             Return False
  100.         Else
  101.             root.Write( stream )
  102.         EndIf
  103.         Return True
  104.     End Method
  105.    
  106.     Rem
  107.     bbdoc: Create an empty IPDoc.
  108.     EndRem
  109.     Function Create:IPDoc( name$="" )
  110.         Local doc:IPDoc = New IPDoc
  111.         doc.root.Name = name
  112.         Return doc
  113.     End Function
  114.    
  115.     Method DataType:TDataType()
  116.         Return gIPDocDataType
  117.     End Method
  118. End Type
  119.  
  120. Rem
  121. bbdoc: A node in an Parse document tree.  A node has functions for reading
  122. EndRem
  123. Type IPNode Extends TData
  124.     Field link:ILink
  125.     Rem
  126.     bbdoc: The name of the node.
  127.     EndRem
  128.     Field Name$
  129.     Rem
  130.     bbdoc: The node's attributes/properties.
  131.     EndRem
  132.     Field Attrs:IList = New IList
  133.     Rem
  134.     bbdoc: The node's children.
  135.     EndRem
  136.     Field Children:IList = New IList
  137.     Rem
  138.     bbdoc: The node's parent node.
  139.     EndRem
  140.     Field Parent:IPNode = Null
  141.    
  142.     ' Useful for spitting out errors when parsing a valid Parse document with bad input
  143.     Field _line%=0
  144.     Field _colm%=0
  145.    
  146.     Rem
  147.     bbdoc: Gets the line number the node was found on.
  148.     EndRem
  149.     Method GetLine%( )
  150.         Return _line
  151.     End Method
  152.    
  153.     Rem
  154.     bbdoc: Gets the column number the node was found on.
  155.     EndRem
  156.     Method GetColumn%( )
  157.         Return _colm
  158.     End Method
  159.    
  160.     Method LineCol$( )
  161.         Return "["+_line+":"+_colm+"]"
  162.     End Method
  163.  
  164.     Rem
  165.     bbdoc: Reads a set of nodes from a stream, optionally passing the amount of bytes to read up to (instead of reading until EOF) to @size.<br/>
  166.     The method will not seek to the beginning of the stream, so if you pass -1 to @size then it will calculate the size of data as @res.Size( ) - @res.Pos( ).<br/>
  167.     Endrem
  168.     Method Read( res:TStream, size%=-1, sline%=1, scolumn%=1 )
  169.         Const E_NAME% = 0
  170.         Const R_NAME% = 2
  171.         Const E_OPEN% = 4
  172.         Const RF_VALUE% = 6
  173.         Const R_COMMENT% = 8
  174.        
  175.         Assert res,"Failed to read file"
  176.  
  177.         If size <= -1 Then size = res.Size( )-res.Pos( )
  178.         Local s% = E_NAME, s_last%
  179.         Local node:IPNode = Self
  180.         Local nf:IPAttr
  181.         Local c%=0
  182.         Local escape%
  183.         Local line%=sline
  184.         Local col%=scolumn
  185.         _line = line
  186.         _colm = col
  187.         Local sz%=0
  188.         Local cfield%=0
  189.         Local lc%
  190.         Local si%=0
  191.        
  192.         While size > 0
  193.             sz = Min(buf.Length, size)
  194.             res.ReadBytes( buf, sz )
  195.             size :- buf.Length
  196.             For Local i:Int = 0 To sz-1
  197.                 lc = c
  198.                 c = buf[i]
  199.                 If c = 13 Or c = 0 Then Continue
  200.                 col :+ 1
  201.                
  202.                 If c = 35 And escape = 0 And s <> R_COMMENT Then
  203.                     If s = R_NAME Then
  204.                         s = E_OPEN
  205.                     ElseIf s = RF_VALUE
  206.                         nf.Content = String.FromShorts( shrts, si )
  207.                         si = 0
  208.                         s = E_NAME
  209.                     EndIf
  210.    
  211.                     s_last = s
  212.                     s = R_COMMENT
  213.                 EndIf
  214.    
  215.                 If c = 10 Then
  216.                     line :+ 1
  217.                     col = 0
  218.                     If s = R_COMMENT Then
  219.                         s = s_last
  220.                     EndIf
  221.                 EndIf
  222.                
  223.               '  DebugLog "["+line+":"+col+":"+s+"] "+c
  224.                
  225.                 If s = R_COMMENT Then Continue
  226.                
  227.                 If c = 10 And escape Then
  228.                     escape = 0
  229.                     Continue
  230.                 EndIf
  231.                
  232.                 If (c = 32 Or c = 9) And (lc = 32 Or lc = 9) Then
  233.                     Continue
  234.                 EndIf
  235.  
  236.                 If c = 92 And escape = 0 Then
  237.                     escape = 1
  238.                     lc = 0
  239.                     Continue
  240.                 EndIf
  241.                
  242.                 Select s
  243.                
  244.                     Case E_NAME
  245.                         If (c = 0 Or c = 10 Or c = 32 Or c = 9) And escape = 0 Then
  246.                             Continue
  247.                         ElseIf c = 123 And escape = 0 Then
  248.                             node = node.AddChild( node.Children.Count( ) )
  249.                             cfield=1
  250.                         ElseIf c = 125 And escape = 0 Then
  251.                             If node = Null Or node = Self Then
  252.                                 memset_( buf, 0, buf.Length )
  253.                                 memset_( shrts, 0, shrts.Length*2 )
  254.                                 Throw "["+line+";"+col+";"+s+"]  Unexpected character '"+Chr(c)+"', cannot close a node when there is none open"
  255.                             EndIf
  256.                             node = node.Parent
  257.                             cfield=1
  258.                         Else
  259.                             shrts[0] = c
  260.                             si = 1
  261.                             s = R_NAME
  262.                             cfield = 1
  263.                         EndIf
  264.                    
  265.                     Case R_NAME
  266.                         If (c = 32 Or c = 9) And escape = 0 Then
  267.                             s = E_OPEN
  268.                         ElseIf c = 123 And escape = 0 Then
  269.                             s = E_NAME
  270.                             node = node.AddChild( String.FromShorts( shrts, si ) )
  271.                             si = 0
  272.                         ElseIf c = 10 And escape = 0 Then
  273.                             cfield = 0
  274.                             s = E_OPEN
  275.                         ElseIf c = 33 And escape = 0 Then
  276.                             node.AddAttr( String.FromShorts( shrts, si ) )
  277.                             si = 0
  278.                             s = E_NAME
  279.                             cfield = 1
  280.                         ElseIf c = 125 And escape = 0 Then
  281.                             memset_( buf, 0, buf.Length )
  282.                             memset_( shrts, 0, shrts.Length*2 )
  283.                             Throw "["+line+";"+col+";"+s+"]  Unexpected character '"+Chr(c)+"', expected name, !, or { -- perhaps you forgot to escape a character?"
  284.                         Else
  285.                             If shrts.Length = si Then shrts = shrts[..shrts.Length*2]
  286.                             shrts[si] = c
  287.                             si :+ 1
  288.                         EndIf
  289.                    
  290.                     Case E_OPEN
  291.                         If c = 123 And escape = 0 Then
  292.                             node = node.AddChild( String.FromShorts( shrts, si ) )
  293.                             si = 0
  294.                             s = E_NAME
  295.                         ElseIf c = 32 Or c = 9 Then
  296.                             Continue
  297.                         ElseIf cfield
  298.                             nf = node.AddAttr( String.FromShorts( shrts, si ) )
  299.                             si = 1
  300.                             shrts[0] = c
  301.                             s = RF_VALUE
  302.                         Else
  303.                             memset_( buf, 0, buf.Length )
  304.                             memset_( shrts, 0, shrts.Length*2 )
  305.                             Throw "["+line+";"+col+";"+s+"]  Unexpected character '"+Chr(c)+"', expected { or attribute value -- perhaps you forgot to escape a character?"
  306.                         EndIf
  307.                    
  308.                     Case RF_VALUE
  309.                         If c = 123 And escape = 0 Then
  310.                             nf.Content = String.FromShorts( shrts, si )
  311.                             si = 0
  312.                             node = node.AddChild( node.Children.Count( ) )
  313.                             s = E_NAME
  314.                             cfield=1
  315.                         ElseIf c = 125 And escape = 0 Then
  316.                             nf.Content = String.FromShorts( shrts, si )
  317.                             si = 0
  318.                             s = E_NAME
  319.                             If node = Null Or node = Self Then
  320.                                 memset_( buf, 0, buf.Length )
  321.                                 memset_( shrts, 0, shrts.Length*2 )
  322.                                 Throw "["+line+";"+col+";"+s+"]  Unexpected character '"+Chr(c)+"', cannot close a node when there is none open"
  323.                             EndIf
  324.                             node = node.Parent
  325.                             cfield=1
  326.                         ElseIf c = 59 And escape = 0 Then
  327.                             nf.Content = String.FromShorts( shrts, si )
  328.                             si = 0
  329.                             s = E_NAME
  330.                             cfield = 1
  331.                         ElseIf c = 10 And escape = 0 Then
  332.                             nf.Content = String.FromShorts( shrts, si )
  333.                             si = 0
  334.                             cfield = 1
  335.                             s = E_NAME
  336.                         Else
  337.                             If escape = 1 Then
  338.                                 Select c
  339.                                     Case 78
  340.                                         c = 10
  341.                                     Case 110
  342.                                         c = 10
  343.                                     Case 48
  344.                                         c = 0
  345.                                     Default
  346.                                 End Select
  347.                             EndIf
  348.                             If shrts.Length = si Then shrts = shrts[..shrts.Length*2]
  349.                             shrts[si] = c
  350.                             si :+ 1
  351.                         EndIf
  352.                    
  353.                 End Select
  354.                
  355.                 escape = 0
  356.             Next
  357.         Wend
  358.        
  359.         If node <> Self Then
  360.             memset_( buf, 0, buf.Length )
  361.             memset_( shrts, 0, shrts.Length*2 )
  362.             Throw "["+line+";"+col+";"+s+"]  Unexpected EOF"
  363.         EndIf
  364.         If s <> E_NAME Then
  365.             Select s
  366.                 Case E_OPEN
  367.                     memset_( buf, 0, buf.Length )
  368.                     memset_( shrts, 0, shrts.Length*2 )
  369.                     Throw "["+line+";"+col+";"+s+"]  Unexpected EOF"
  370.                 Case RF_VALUE
  371.                     nf.Content = String.FromShorts( shrts, si )
  372.                 Case E_NAME
  373.                 Case R_COMMENT
  374.             End Select
  375.         EndIf
  376.        
  377.         memset_( buf, 0, buf.Length )
  378.         memset_( shrts, 0, shrts.Length*2 )
  379.     End Method
  380.    
  381.    Rem
  382.    bbdoc: Writes the node, its children, and its Attrs to a stream in the order of <i>[name] { [Attrs] [children] }</i>
  383.    EndRem
  384.     Method Write( out:TStream, indent$="" )
  385.         Local attrIndent$ = indent
  386.         If Parent Then ' Nodes without parents are assumed to be documents
  387.             out.WriteLine( indent+Name.Replace("","\").Replace(" "," ").Replace("#","#").Replace("{","{").Replace("}","}").Replace("!","!").Replace("~t","~t").Replace("~n","
  388. ")+" {" )
  389.             attrIndent :+ "    "
  390.         EndIf
  391.  
  392.         For Local i:IPAttr = EachIn Attrs
  393.             If i.Content.Length = 0 Then
  394.                 out.WriteLine( attrIndent+i.Name.Replace("","\").Replace(" "," ").Replace("#","#").Replace("{","{").Replace("}","}").Replace("!","!").Replace("~t","~t").Replace("~n","
  395. ")+"!" )
  396.             Else
  397.                 Local outp$ = attrIndent+i.Name.Replace("","\").Replace(" "," ").Replace("#","#").Replace("{","{").Replace("}","}").Replace("!","!").Replace("~t","~t").Replace("~n","
  398. ")
  399.                 outp :+ " "+i.Content.Replace("","\").Replace(" "," ").Replace("#","#").Replace("{","{").Replace("}","}").Replace("~n","~n")
  400.                 out.WriteLine( outp )
  401.             EndIf
  402.         Next
  403.         For Local i:IPNode = EachIn Children
  404.             i.Write( out, indent+"    " )
  405.         Next
  406.        
  407.         If Parent Then
  408.             out.WriteLine( indent+"}" )
  409.         EndIf
  410.     End Method
  411.    
  412.     Rem
  413.     bbdoc: Prepares the node, its children, and its Attrs for collection by removing itself from its parent and disposing of its children and Attrs.
  414.     EndRem
  415.     Method Dispose( )
  416.         Name = Null
  417.         Parent = Null
  418.         If link Then link.Remove( )
  419.         link = Null
  420.         If Attrs Then
  421.             For Local i:IPAttr = EachIn Attrs
  422.                 i.Dispose( )
  423.             Next
  424.             Attrs.Clear( )
  425.         EndIf
  426.         Attrs = Null
  427.         If Children Then
  428.             For Local i:IPNode = EachIn Children
  429.                 i.Dispose( )
  430.             Next
  431.             Children.Clear( )
  432.         EndIf
  433.         Children = Null
  434.     End Method
  435.  
  436.     Rem
  437.     bbdoc: Adds a child to the node with the name @fname and returns it.
  438.     returns: The new @IPNode.
  439.     EndRem
  440.     Method AddChild:IPNode( fname$ )
  441.         Local i:IPNode = New IPNode
  442.         i.Name = fname
  443.         i.link = Children.AddLast( i )
  444.         i.Parent = Self
  445.         Return i
  446.     End Method
  447.  
  448.     Rem
  449.     bbdoc: Adds an attribute to the node with the name @fname and an optional @value and returns it.
  450.     returns: The new @IPAttr.
  451.     EndRem
  452.     Method AddAttr:IPAttr( fname$, value$="" )
  453.         Local i:IPAttr = New IPAttr
  454.         i.Name = fname
  455.         i.Content = value
  456.         i.link = Attrs.AddLast( i )
  457.         i.Parent = Self
  458.         Return i
  459.     End Method
  460.    
  461.     Rem
  462.     bbdoc: Gets an attribute and, if you pass a value to @fDefaultValue, will create the new attribute if one does not already exist.<br/>
  463.     This method makes the assumption that all attribute names in the node are unique.
  464.     returns: A string containing the contents of the attribute.
  465.     EndRem
  466.     Method GetAttr$( fname$, fDefaultValue$=Null )
  467.         Local f:IPAttr = FindAttr( fname )
  468.         If Not f And fDefaultValue <> Null Then
  469.             AddAttr( fname, fDefaultValue )
  470.             Return fDefaultValue
  471.         ElseIf f
  472.             Return f.content
  473.         EndIf
  474.         Return Null
  475.     End Method
  476.    
  477.     Rem
  478.     bbdoc: Sets an attribute with the name @fname to @value.  If the attribute doesn't exist, it is created.
  479.     EndRem
  480.     Method SetAttr( fname$, fvalue$ )
  481.         Local f:IPAttr = FindAttr( fname )
  482.         If Not f Then
  483.             AddAttr( fname, fvalue )
  484.         Else
  485.             f.Content = fvalue
  486.         EndIf
  487.     End Method
  488.    
  489.     Rem
  490.     bbdoc: Finds a child node with the name @fname.  If children with the same name exist, you can iterate over them by passing the last-found child to @last.
  491.     returns: The requested IPNode if successful, otherwise Null.
  492.     EndRem
  493.     Method FindNode:IPNode( fname$, last:IPNode = Null )
  494.         If last = Null Then
  495.             last = IPNode( Children.GetFirst( ) )
  496.         Else
  497.             If Not last.link.NextLink( ) Then
  498.                 Return Null
  499.             EndIf
  500.             last = IPNode( last.link.NextLink( ).Value( ) )
  501.         EndIf
  502.  
  503.         While ( last <> Null )
  504.             If last.name = fname Return last
  505.             If Not last.link.NextLink( ) Then
  506.                 Return Null
  507.             EndIf
  508.             last = IPNode( last.link.NextLink( ).Value( ) )
  509.         Wend
  510.  
  511.         Return Null
  512.     End Method
  513.  
  514.     Rem
  515.     bbdoc: Finds an attribute with the name @fname.  If attributes with the same name exist, you can iterate over them by passing the last-found attribute to @last.<br/>
  516.     This function will return the IPAttr, not a string like GetAttr.
  517.     returns: The requested IPNode if successful, otherwise Null.
  518.     EndRem
  519.     Method FindAttr:IPAttr( fname$, last:IPAttr = Null )
  520.         If last = Null Then
  521.             last = IPAttr( Attrs.GetFirst( ) )
  522.         Else
  523.             If Not last.link.NextLink( ) Then
  524.                 Return Null
  525.             EndIf
  526.             last = IPAttr( last.link.NextLink( ).Value( ) )
  527.         EndIf
  528.  
  529.         While ( last <> Null )
  530.             If last.name = fname Return last
  531.             If Not last.link.NextLink( ) Then
  532.                 Return Null
  533.             EndIf
  534.             last = IPAttr( last.link.NextLink( ).Value( ) )
  535.         Wend
  536.  
  537.         Return Null
  538.     End Method
  539.    
  540.     Method DataType:TDataType( )
  541.         Return gIPNodeDataType
  542.     End Method
  543. End Type
  544.  
  545. Rem
  546. bbdoc: Attribute class for an IPNode.
  547. EndRem
  548. Type IPAttr
  549.     Rem
  550.     bbdoc: Name of the attribute.
  551.     EndRem
  552.     Field Name$
  553.     Rem
  554.     bbdoc: Content/value of the attribute.
  555.     EndRem
  556.     Field Content$
  557.     Rem
  558.     bbdoc: The parent/owner (node) of the attribute.
  559.     EndRem
  560.     Field Parent:IPNode
  561.    
  562.     Field link:ILink
  563.  
  564.     Field _line%=0
  565.     Field _colm%=0
  566.    
  567.     Rem
  568.     bbdoc: Gets the line number the attribute was found on.
  569.     EndRem
  570.     Method GetLine%( )
  571.         Return _line
  572.     End Method
  573.    
  574.     Rem
  575.     bbdoc: Gets the column number the attribute was found on.
  576.     EndRem
  577.     Method GetColumn%( )
  578.         Return _colm
  579.     End Method
  580.    
  581.     Method LineCol$( )
  582.         Return "["+_line+":"+_colm+"]"
  583.     End Method
  584.    
  585.     Rem
  586.     bbdoc: Prepares the attribute for collection by removing itself from its parent.
  587.     EndRem    
  588.     Method Dispose( )
  589.         If link Then link.Remove( )
  590.         link = Null
  591.         Parent = Null
  592.         Content = Null
  593.         Name = Null
  594.     End Method
  595. End Type


Comments :


N(Posted 1+ years ago)

 Updated source to include changes since last submission.  Mainly code clean-up.# Now uses Cower.Collections (aka collections.bmx)- to change to Brl.LinkedList, replace /(:)?I(List|Link)/ with /$1T$2/ and change methods to use BRL's similar/same methods.# Added IPDoc type as a means of managing the collection of nodes more easily.  Documents are never referenced by nodes, so once the document is collected, the root node is disposed and will, presumably, be collected afterward.# Auxiliary functions removed (reason being, if you cannot yet grasp OO coding, you should not be using this code).


 

SimplePortal 2.3.6 © 2008-2014, SimplePortal