September 17, 2019, 08:26:01 AM

Author Topic: JSON parser for NG?  (Read 917 times)

Offline Yellownakji

  • Full Member
  • ***
  • Posts: 247
  • No.
JSON parser for NG?
« on: April 06, 2019, 11:16:59 PM »
Is there a module for JSON support?  I've seen modules like "rjson.mod", but they're not NG compatible; The latest BMK has a fit.

If there is a mod, it would save me a lot of time.  Otherwise, i'm just gonna make a JSON destroyer because i'm NOT writing a JSON parser. *barfs* ::)

Offline Derron

  • Hero Member
  • *****
  • Posts: 2402
Re: JSON parser for NG?
« Reply #1 on: April 07, 2019, 06:22:28 AM »
Brl.json (NG)
https://github.com/bmx-ng/brl.mod/tree/master/json.mod

You should have it if you use an updated BlitzMax NG installation.


Bye
Ron

Offline Yellownakji

  • Full Member
  • ***
  • Posts: 247
  • No.
Re: JSON parser for NG?
« Reply #2 on: April 07, 2019, 09:49:15 PM »
Brl.json (NG)
https://github.com/bmx-ng/brl.mod/tree/master/json.mod

You should have it if you use an updated BlitzMax NG installation.


Bye
Ron

Thanks for this.   No, i only really have the modules in need.   I went to the "modules" sticky, but i didn't see JSON anywhere, except RJSON which is broken.  I'll be sure to look in the default modules from now on, first.   I ended up writing a destroyer, however.

Thanks.

Offline Matty

  • Hero Member
  • *****
  • Posts: 730
    • MattiesGames
Re: JSON parser for NG?
« Reply #3 on: April 08, 2019, 02:33:10 AM »
Interesting...I just had to write a json parser for work in a language that didn't currently accept json files (only reads standard text files)....took me 7 hours from design on paper to implementation as an 'include'...not blitzmax code so not really helpful...but still..working on this at the same time as you asked about it.


Offline Yellownakji

  • Full Member
  • ***
  • Posts: 247
  • No.
Re: JSON parser for NG?
« Reply #4 on: April 08, 2019, 03:29:33 AM »
Interesting...I just had to write a json parser for work in a language that didn't currently accept json files (only reads standard text files)....took me 7 hours from design on paper to implementation as an 'include'...not blitzmax code so not really helpful...but still..working on this at the same time as you asked about it.

Yeah, based on how JSON is formatted, i knew it would be some time before i had an ACTUAL parser written and i didn't see any, at the time, for blitzmax.   I ended up writing a destroyer, which takes a lot of the syntax crap, converts it to my own method and then spits out what i want.

Offline Hezkore

  • Jr. Member
  • **
  • Posts: 79
  • 🍑 Juicy
Re: JSON parser for NG?
« Reply #5 on: August 03, 2019, 06:03:52 PM »
I'm sorry to necro this thread.
But I'm trying to use BRL.Json, but there's no documentation on it.
I'm having some issues actually loading a Json from string.

Any examples floating around?
Keep it simple.

Offline Hezkore

  • Jr. Member
  • **
  • Posts: 79
  • 🍑 Juicy
Re: JSON parser for NG?
« Reply #6 on: August 03, 2019, 06:48:08 PM »
Okay so I found some examples.
But it's pretty "wonky" overall.
For example, reading through the Telegram API Json; it errors out with "Unhandled Exception:Unicode character out of UCS-2 range" on any message that contains an emoji... which... is a lot of messages heh.
Code: [Select]
"text":"test \ud83e\udd37\u200d\u2642\ufe0f"
Any tips?
Keep it simple.

Offline Derron

  • Hero Member
  • *****
  • Posts: 2402
Re: JSON parser for NG?
« Reply #7 on: August 03, 2019, 10:13:47 PM »
That is not a brl.json error but a brl.blitz error:
bbStringFromUTF8String in blitz_string.c throws this.

alternatively the same for brl.stringbuilder:
bmx_stringbuilder_append_utf8string in glue.c


Maybe rising an issue on github (github.com/bmx-ng/brl.mod) helps?


Edit: Maybe the data you receive is not utf-8 but utf-16 or utf-32. The BlitzMax functions only handle utf-8.
Edit2: is the text in the codebox what you receive - or is this something which the forum here "converted" to display properly with the used charset of the forum theme?

bye
Ron

Offline TomToad

  • Sr. Member
  • ****
  • Posts: 428
Re: JSON parser for NG?
« Reply #8 on: August 04, 2019, 03:38:30 AM »
BlitzMax strings only use code points 0-65535. Emojis are non standard and start at code point 65536 and above, hence the error. I had the same problem when parsing Xml with emojis. Had to rewrite the utf-8 to unicode routine so instead of an error, it would just replace the emojis with an"?".
------------------------------------------------
8 rabbits equals 1 rabbyte.

Offline Yellownakji

  • Full Member
  • ***
  • Posts: 247
  • No.
Re: JSON parser for NG?
« Reply #9 on: August 04, 2019, 08:49:26 AM »
Okay so I found some examples.
But it's pretty "wonky" overall.
For example, reading through the Telegram API Json; it errors out with "Unhandled Exception:Unicode character out of UCS-2 range" on any message that contains an emoji... which... is a lot of messages heh.
Code: [Select]
"text":"test \ud83e\udd37\u200d\u2642\ufe0f"
Any tips?

I ended up writing my own parser.   I'm sick of relying on others' works, only to have them be not compatible with NG or just poorly documented to the point of no return. 

BRL.JSON in general is pure crap, so i wasn't going to stick with it in regardless.   Sorry i cannot offer any tips;  best i can say is to ditch it.

JSON follows a very strict bracket system, so it's really easy to write your own interpreter.   Check for commas, create new lists based on bracket formations.   You'll get the idea once you really start to compare JSONs together.

Offline Derron

  • Hero Member
  • *****
  • Posts: 2402
Re: JSON parser for NG?
« Reply #10 on: August 04, 2019, 09:13:26 AM »
It is your opinion to say Brucey's brl.json is crap.
Feel free to add your code so the community can benefit from your less "crap", better performing, better documented and overall better module.


I go along with that it needs examples and maybe some more documentation - but this is what the NG users could do for Brucey - so he does not need to take care of this alongside to other stuff.


@ own parser
So how did you solve the UTF8-handling which is broken in "brl.blitz" and not in "brl.json". Let me guess, you wrote your own code doing it superior but you hide it from us so only you can benefit?


So please: give your oo json implementation which covers the stuff brl.json covers and handles. Maybe make "persistence.mod" work with it too so it is far easier for us to serialize blitzmax objects into json data - and vice versa. For now there is just brl.json doing this and of course a more lightweight, less error prone and better performing lib is always appreciated.


bye
Ron

Offline TomToad

  • Sr. Member
  • ****
  • Posts: 428
Re: JSON parser for NG?
« Reply #11 on: August 04, 2019, 10:06:59 AM »
The line that throws the error is in brl.mod/blitz.mod/blitz_string.c
This line checks if the utf-8 code point is within 0-65535 range and throws an error if not.

You can either replace the value with a placeholder, replace the emoji with the ASCII equivalent, or avoid using emojis in your json file altogether.

Eta: or write your own string from utf-8 function and update brl.mod/json.mod/glue.c replace all bbStringFromUTF8String calls with your own function.
------------------------------------------------
8 rabbits equals 1 rabbyte.

Offline Derron

  • Hero Member
  • *****
  • Posts: 2402
Re: JSON parser for NG?
« Reply #12 on: August 04, 2019, 10:55:39 AM »
And 65535 is "2^16 - 1" so something which is the biggest number supported by (unsigned) shorts.
And shorts are used in the string functions. Dunno what impact it would have to replace it with a bigger sized number type. For other stuff there is a "bbStringFromLong" (and others) but the "fromutf8" variant relies on the shorts.


https://github.com/openssl/openssl/blob/master/crypto/asn1/a_utf8.c
int UTF8_getc(const unsigned char *str, int len, unsigned long *val) might be worth a look.


Created an issue for this:
https://github.com/bmx-ng/brl.mod/issues/149

bye
Ron

Offline Hezkore

  • Jr. Member
  • **
  • Posts: 79
  • 🍑 Juicy
Re: JSON parser for NG?
« Reply #13 on: August 04, 2019, 02:31:19 PM »
I don't mind the Json module for NG. Once I figured out how it worked; it was pretty easy.
Though I'm thinking about extending it with a "TEasyJson" which makes it even easier, heh.
But yeah, examples and documentation is needed! (also the most boring part to make haha)

The Unicode string issue is bugging me though.
Code: [Select]
"text":"test \ud83e\udd37\u200d\u2642\ufe0f"^ is what Firefox is telling me that the Json I'm getting from the Telegram API contains.
So I'm not sure how it all works, or if it's converted by Firefox.
I'm just calling "getUpdates" on my Telegram Bot - https://core.telegram.org/bots/api#getupdates
Keep it simple.

Offline Hezkore

  • Jr. Member
  • **
  • Posts: 79
  • 🍑 Juicy
Re: JSON parser for NG?
« Reply #14 on: August 04, 2019, 02:41:27 PM »
I've made a small example.
If you just make a string and input an emoji; it works.
But if you use some of those fancy From*String functions; it fails.
So I'm guessing the Json parser uses one of those?

Code: [Select]
SuperStrict
Framework BRL.StandardIO

'Local emojiString:String =  String.FromUTF8String( "💩" ) ' < Doesn't work
Local emojiString:String =  "💩" ' < Works

' Opening the result in Sublime shows the emoji
SaveString( emojiString, "tmp.txt"  )

Might be old news to everyone but me, but hey... it makes things clearer for me at least. ¯\_(ツ)_/¯
Keep it simple.