SyntaxBomb - Indie Coders

I have been really struggling with what I expected to be a straightforward problem :
Downloading of html from a given URL

I had been using the following (( some dependencies for constants and filepath handling etc. are not included )) in B3D / B+ but was unabble to convert to BMax because of how BMax uses the Sockets etc.

;These provide a set of control codes. Typically the entry would be akin to " or &#x26 etc.
Const HTM_CODE_LOWER_PREFIX$="&#x"
Const HTM_CODE_LOWER_SUFFIX$=";"

Const HTM_CARRIAGE_RETURN$="<br>"

Const HTM_STANDARDSYMBOLS_PREFIX$="<"
Const HTM_STANDARDSYMBOLS_SUFFIX$=">"

Const HTM_ENCODED_URL_CHARACTERS$="!£$^*()-=[]{}¬`;:'@#~|<>,?"

; FIXES CONTROL CODES USED BY WEBPAGE HTML
Function HTM_CleanControlCodesText$(sText$)
DebugLog("HTM (Clean HTM): Cleaning HTML Control Codes")

; Non-Standard symbolisations
Local IterFixes

Local SeekEnd=0
Local SeekStart=0
Local SeekWidth=0

Local MidString$=""

; Fix for typical html control codes with <> tags
SeekStart=Instr(sText,HTM_STANDARDSYMBOLS_PREFIX)
If (SeekStart)
While (SeekStart)
SeekEnd=Instr(sText,HTM_STANDARDSYMBOLS_SUFFIX,SeekStart)
SeekWidth=(SeekEnd-SeekStart)

If (SeekStart*SeekEnd)
MidString=Mid(sText,SeekStart,SeekWidth+1)
DebugLog("HTM (Clean HTM): Removed instances of code: "+Chr(34)+MidString+Chr(34))
sText=Replace(sText,MidString,"")
End If
SeekStart=Instr(sText,HTM_STANDARDSYMBOLS_PREFIX)
Wend
End If

; FIX FOR ATYPICAL CONTROL CODES FOR SYMBOLS WITH CONVERTED HEX NOTATIONS (i.e. &#xnn )
SeekStart=Instr(Lower(sText),HTM_CODE_LOWER_PREFIX)
If (SeekStart)
While (SeekStart)
SeekEnd=Instr(Lower(sText),HTM_CODE_LOWER_SUFFIX,SeekStart)
SeekWidth=(SeekEnd-SeekStart)
If (SeekStart*SeekEnd)
MidString=Mid(sText,SeekStart,SeekWidth+1)
DebugLog("HTM (Clean HTM): Removed instances of code: "+Chr(34)+MidString+Chr(34))
sText=Replace(sText,MidString,"")
End If
SeekStart=Instr(Lower(sText),HTM_CODE_LOWER_PREFIX)
Wend
End If

; Remove any remaining ascii control codes
DebugLog("HTM (Clean HTM): Removing ASCII code instances")
Local IterAsc
For IterAsc=0 To 31
sText=Replace(sText,Chr(IterAsc),"")
Next

Return sText$
End Function

;ENCODES CHARACTERS IN A STRING SUITABLE FOR URL ENCODING
Function HTM_EncodeURL$(sText$)

DebugLog("HTM (Fix url): Encoding "+Chr(34)+sText+Chr(34))

sText=Replace(Lower(Trim(sText)),"_","+")
sText=Replace(Lower(Trim(sText))," ","+")
sText=Replace(sText,"&","+")
sText=Replace(sText,STRING_PERCENT,"percent")
sText=Replace(sText,Chr(34),"")

Local Count
Local Char$
For Count=1 To Len(sText)
Char=(Mid(sText,Count,1))
Replace(sText,Char,HTM_URLEncode(Char))
Next

;Deal with multiple +'s
While Instr(sText,"++")
sText=(Replace(sText,"++","+"))
Wend

sText=Lower(Trim(sText))

DebugLog("HTM (Fix url): Result of encoding: "+Chr(34)+sText+Chr(34))

Return sText

End Function

;ENCODES NON-ALPHANUMERIC CHARACTERS WITH VALID URL HTML CODES
Function HTM_URLEncode(s_Char)
If (Not(GetIsAlphanumericCharacter(s_Char)))
If (Instr(HTM_ENCODED_URL_CHARACTERS,s_Char))
s_Char=STRING_PERCENT+Str(Asc(s_Char))
End If
End If
Return s_Char
End Function
;~IDEal Editor Parameters:
;~C#Blitz+

;________________________________________________________________________________________________________________

Const NET_PROTOCOL_HTTP$="http"
Const NET_PROTOCOL_HTTPS$="https"
Const NET_PROTOCOL_FTP$="ftp"
Const NET_WORLDWIDEWEB$="www"

Const NET_PROTOCOL_HANDLE_HTTP09$="HTTP/0.9"
Const NET_PROTOCOL_HANDLE_HTTP10$="HTTP/1.0"
Const NET_PROTOCOL_HANDLE_HTTP11$="HTTP/1.1"

Const NET_VALID_HEX$="123456789ABCDEF"

Const NET_USERAGENT$="EKD_Engine_Agent"
Const NET_BROWSERAGENT_IE11$="Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko"
Const NET_BROWSERAGENT_IE8$="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2)"
Const NET_BROWSERAGENT_IE6$="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"

Const NET_PROTOCOL_SUFFIX$="://";cs_STRING_COLON+cs_STRING_WEB_SEPARATOR+cs_STRING_WEB_SEPARATOR

Const NET_DLE_FLAGS=7

Const NET_DLE_COM=1
Const NET_DLE_UK=2
Const NET_DLE_ORG=4

Const NET_DOMAIN_COM$="com"
Const NET_DOMAIN_UK$="co.uk"
Const NET_DOMAIN_ORG$="org"

Const NET_LOCAL_SEND_GET$="GET"
Const NET_LOCAL_SEND_POST$="POST"

Const NET_INVALIDFILECHARS$="=*:?<>|"

Const NET_SERVER_MESSAGE_REDIRECT$="location"
Const NET_SERVER_MESSAGE_LENGTH$="content-length"
Const NET_SERVER_MESSAGE_ENCODING$="transfer-encoding"
Const NET_SERVER_MESSAGE_CHUNKED$="chunked"

Const NET_PORT_DEFAULT_HTTP_PORT=80
Const NET_PORT_DEFAULT_HTTPS_PORT=443

Const NET_RESPONSE_OK=200
Const NET_RESPONSE_OK_CREATE=201
Const NET_RESPONSE_OK_ACCEPT=202
Const NET_RESPONSE_OK_PARTIAL=203
Const NET_RESPONSE_OK_NOREPLY=204

Const NET_RESPONSE_OK_MESSAGE=220

Const NET_RESPONSE_BAD_REQUEST=400
Const NET_RESPONSE_BAD_UNAUTHORISED=401
Const NET_RESPONSE_BAD_PAYMENT=402
Const NET_RESPONSE_BAD_FORBIDDEN=403
Const NET_RESPONSE_BAD_NOTFOUND=404

Const NET_RESPONSE_ERR_SERVER=500
Const NET_RESPONSE_ERR_NOTIMPLEMENTED=501
Const NET_RESPONSE_ERR_BANDWIDTH=502
Const NET_RESPONSE_ERR_TIMEOUT=503

Const NET_RESPONSE_REDIR_MOVED=301
Const NET_RESPONSE_REDIR_FOUND=302
Const NET_RESPONSE_REDIR_METHOD=303
Const NET_RESPONSE_REDIR_NOCHANGE=304

Const NET_RESPONSE_QUERY_PASS=331

Const NET_RESPONSE_ACTION_ERROR=0
Const NET_RESPONSE_ACTION_DOWNLOAD=1
Const NET_RESPONSE_ACTION_REDIRECT=2

Const NET_CHUNKSIZE=4096

Const NET_TIMEOUT_MILLISECS=5000

 ; WRITES BYTES FROM STREAM TO FILE
Function NET_DownloadFromStream(hst_Stream, DownloadPath$)
Local n4_ContentLength=-1
Local nb_Chunked = False

Local s_Line$

Local ns_Pos
Local s_Part1$
Local s_Part2$

;Recurse Bytes from stream...
Repeat

s_Line$ = Trim(NET_ReadStreamLine$(hst_Stream))

DebugLog("NET (DownloadFromStream): Server: "+s_Line)

If (s_Line$ = "")
;END OF STREAM!
DebugLog("NET (DownloadFromStream): Null data line from server")
Exit
End If

ns_Pos = Instr(s_Line,":")
s_Part1 = Trim(Left(s_Line, ns_Pos - 1))
s_Part2 = Trim(Right(s_Line, Len(s_Line) - ns_Pos))

Select (Lower(s_Part1$))
Case NET_SERVER_MESSAGE_LENGTH:
n4_ContentLength = Int(s_Part2)
Case NET_SERVER_MESSAGE_ENCODING:
If (Lower(s_Part2$) = Lower(NET_SERVER_MESSAGE_CHUNKED))
nb_Chunked = True
End If
End Select

Until (s_Line="")

If (Eof(hst_Stream)<0)
DebugLog("NET (DownloadFromStream): Error in processing stream. Aborting process")
NET_CloseTCPConnection(hst_Stream)
Return False
End If

Local hst_File=False

hst_File = WriteFile(DownloadPath$)

Local mb_Bank
Local n4_Available

If (Not(n4_ContentLength))
DebugLog("NET (DownloadFromStream): Data length not provided")

CloseFile hst_File
hst_File=False

NET_CloseTCPConnection(hst_Stream)

Return (FileType(DownloadPath)=1)

Else

DebugLog("NET (DownloadFromStream): Retreiving data form server...")

If (Not(nb_Chunked))

DebugLog("NET (DownloadFromStream): Data not served in chunked format")

mb_Bank = CreateBank(NET_CHUNKSIZE)
ns_Pos = 0

Repeat
n4_Available = (n4_ContentLength - ns_Pos)

If (n4_Available > NET_CHUNKSIZE)
ReadBytes mb_Bank, hst_Stream, 0, NET_CHUNKSIZE
WriteBytes mb_Bank, hst_File, 0, NET_CHUNKSIZE
ns_Pos = ns_Pos + NET_CHUNKSIZE

Else

ReadBytes mb_Bank, hst_Stream, 0, n4_Available
WriteBytes mb_Bank, hst_File, 0, n4_Available

ns_Pos=n4_Available
Exit
EndIf

Until ns_Pos=0

FreeBank mb_Bank

CloseFile hst_File
NET_CloseTCPConnection(hst_Stream)
hst_File=False

Return (FileType(DownloadPath)=1)
End If

If (nb_Chunked)
DebugLog("NET (DownloadFromStream): Data served in chunked format")

If (Not(hst_File))
hst_File =WriteFile(DownloadPath)
End If

mb_Bank = CreateBank(NET_CHUNKSIZE)
Local ns_IterBits

Repeat

s_Line$ = Trim(Upper(NET_ReadStreamLine$(hst_Stream)))

ns_Pos = 0

For ns_IterBits = 1 To Len(s_Line$)
ns_Pos = 16 * ns_Pos + Instr(NET_VALID_HEX,Mid$(s_Line, ns_IterBits, 1))
Next

If (Not(ns_Pos))
Exit
End If

If (BankSize(mb_Bank) < ns_Pos)
ResizeBank mb_Bank, ns_Pos
End If

ReadBytes mb_Bank, hst_Stream, 0, ns_Pos
WriteBytes mb_Bank, hst_File, 0, ns_Pos

ReadShort(hst_Stream)

Until ns_Pos=0

FreeBank mb_Bank

CloseFile hst_File
hst_File=False

NET_CloseTCPConnection(hst_Stream)
Return (FileType(DownloadPath)=1)
Else
DebugLog("NET (DownloadFromStream): Data served in invalid format")

NET_CloseTCPConnection(hst_Stream)

CloseFile hst_File
hst_File=False

Return False
EndIf
End If
End Function

;Function NET_ReadStreamBytes(hst_StreamHandle,hmb_BankHandle,n4_ByteCount=1,n4_Offset=0)
; If (Not(hst_StreamHandle))
; DebugLog("NET (ReadBytes: Stream Handle is null")
; Return
; End If
;
; WriteBytes(hmb_BankHandle,hst_StreamHandle,n4_Offset,n4_ByteCount)
;
;End Function

;RETURNS LINE STRING READ FROM STREAM
Function NET_ReadStreamLine$(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadLinet): Stream Handle is null")
Return ""
End If

Local s_Line$=ReadLine(hst_StreamHandle)
Return s_Line$
End Function

;RETURNS STRING READ FROM STREAM
Function NET_ReadStreamString$(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadString): Stream Handle is null")
Return ""
End If

Local s_String$=ReadString(hst_StreamHandle)

Return s_String

End Function

;RETURNS 4-BYTE INTEGER READ FROM STREAM
Function NET_ReadStreamInt%(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadInt): Stream Handle is null")
Return 0
End If

Local n4_Int$=ReadInt(hst_StreamHandle)

Return n4_Int

End Function

;RETURNS 2-BYTE INTEGER READ FROM STREAM
Function NET_ReadStreamShort(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadShort): Stream Handle is null")
Return 0
End If

Local ns_Short=ReadShort(hst_StreamHandle)

Return ns_Short

End Function

;RETURNS 1-BYTE INTEGER READ FROM STREAM
Function NET_ReadStreamByte(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadByte): Stream Handle is null")
Return 0
End If
Local m_Byte=ReadByte(hst_StreamHandle)

Return m_Byte
End Function

;RETURNS 4-BYTE FLOATING POINT VALUE READ FROM STREAM
Function NET_ReadStreamFloat#(hst_StreamHandle)
If (Not(hst_StreamHandle))
DebugLog("NET (ReadFloat): Stream Handle is null")
Return 0.0
End If

Local f_Float#=ReadFloat(hst_StreamHandle)

Return f_Float#

End Function

; SIMPLIFIES HTTP SERVER RESPONSE
Function NET_ResolveServerResponse(hst_StreamHandle)

If (Not(hst_StreamHandle))
DebugLog("NET (Resolve Response): Null stream handle, response resolved as bad")
Return NET_RESPONSE_BAD_REQUEST
End If

Local sLine$ = ReadLine(hst_StreamHandle)

DebugLog("NET (Resolve Response): Response "+Chr(34)+sLine+Chr(34))

If (sLine="")
DebugLog("NET (Resolve Response): No response message from server")
Return NET_RESPONSE_BAD_REQUEST
End If

Local ns_Pos1 = Instr(sLine$," ")
Local ns_Pos2 = Instr(sLine$," ", ns_Pos1 + 1)

If (Not(ns_Pos1*ns_Pos2))
DebugLog("NET (Resolve Response): Incorrect message format from server")
Return NET_RESPONSE_BAD_REQUEST
End If

Return Int(Mid(sLine$,ns_Pos1, ns_Pos2 - ns_Pos1))
End Function

; WHEN ENCOUNTERING 3XX RESPONSE FROM SERVER (CONTENT REDIRECTION), NEXT SERVER TRANSMISSION SHOULD BE REDIRECTION PATH
Function NET_LocateRedirectionPathFromStream$(hst_Stream)
Local s_Line$

Local Redir$
Local Pos

Local snt_NewPath$

Repeat
s_Line$ = Trim(ReadLine(hst_Stream))

DebugLog("NET (Redirect): Server: "+s_Line)

If (s_Line="")
Exit
End If

Pos = Instr(s_Line$,":")

If (Pos)
If (Lower(Trim(Left(s_Line$,Pos-1)))=Lower(NET_SERVER_MESSAGE_REDIRECT))
snt_NewPath = FixPath(Trim(Right(s_Line$, Len(s_Line$) - Pos)),False,True)
DebugLog("NET (Redirect): Obtained redirected path for content: "+snt_NewPath)
DebugLog("NET (Redirect): Closing current stream to commence new download from relocation.")
NET_CloseTCPConnection(hst_Stream)

Return snt_NewPath$
End If
End If

Until s_Line=""

DebugLog("NET (Redirect): Unable to obtain redirection path. Download aborted")
NET_CloseTCPConnection(hst_Stream)

Return ""
End Function

; RETURNS DOMAIN NAME AND DLE ONLY
Function NET_RetrieveDomain$(snt_Path$)
Local s_Sectors$=NET_TrimProtocolHead(snt_Path)
snt_Path$ = Left(s_Sectors,Instr(s_Sectors,"/")-1)

Return snt_Path
End Function

; OPENS TCP CONNECTION STREAM TO DOWNLOAD PATH AND RETURNS HANDLE
Function NET_CreateTCPConnection(snt_FullServerPath$)
snt_FullServerPath=FixPath(snt_FullServerPath,False,True)

Local s_Protocol$=NET_GetProtocolPrefix$(snt_FullServerPath$)
Local ns_Port=NET_GetProtocolPort(snt_FullServerPath)
Local snt_Domain$=NET_RetrieveDomain(snt_FullServerPath)
DebugLog("NET (Create TCP): Connection Polling "+snt_Domain+" via port "+Str(ns_Port))
;Open TCP stream
Local hst_tcp = OpenTCPStream(snt_Domain$, ns_Port )
DebugLog("NET (Create TCP): Connection 0x"+Str(hst_tcp)+" opened to "+DottedIP(TCPStreamIP(hst_tcp))+":"+Str(ns_Port))
Return hst_tcp
End Function

; CLOSES TCP STREAM CONNECTION
Function NET_CloseTCPConnection(hst_TCP=False)
If (hst_TCP)
DebugLog("NET (Close TCP): Stream 0x"+Str(hst_TCP)+" to "+DottedIP(TCPStreamIP(hst_TCP))+":"+Str(TCPStreamPort( hst_TCP ))+" closed")
CloseTCPStream(hst_TCP)
Else
DebugLog("NET (Close TCP): Attempted To close Null stream")
End If
End Function

;RETUIRNS A SIMPLIFIED VALUE DETERMINING APPROPRIATE ACTION ACCORDING TO RESPONSE VALUE
Function NET_GetActionByServerResponse(ns_ServerResponseCode=0)
Select (ns_ServerResponseCode)
Case NET_RESPONSE_OK, NET_RESPONSE_OK_ACCEPT, NET_RESPONSE_OK_CREATE, NET_RESPONSE_OK_MESSAGE, NET_RESPONSE_OK_NOREPLY, NET_RESPONSE_OK_PARTIAL:
DebugLog("NET (GetAction): Permission granted: Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_DOWNLOAD
Case NET_RESPONSE_REDIR_MOVED,NET_RESPONSE_REDIR_FOUND,NET_RESPONSE_REDIR_METHOD, NET_RESPONSE_REDIR_NOCHANGE:
DebugLog("NET (GetAction): Destination redirected: Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_REDIRECT
Case NET_RESPONSE_BAD_NOTFOUND:
DebugLog("NET (GetAction): Destination not found : "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_ERROR
Case NET_RESPONSE_BAD_FORBIDDEN,NET_RESPONSE_BAD_REQUEST,NET_RESPONSE_BAD_PAYMENT,NET_RESPONSE_BAD_UNAUTHORISED:
DebugLog("NET (GetAction): Permission refused: Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_ERROR
Case NET_RESPONSE_ERR_TIMEOUT:
DebugLog("NET (GetAction): Attempt timed-out: Response code: "+Str(ns_ServerResponseCode))
Case NET_RESPONSE_ERR_BANDWIDTH:
DebugLog("NET (GetAction): Server busy: Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_ERROR
Case NET_RESPONSE_ERR_NOTIMPLEMENTED,NET_RESPONSE_ERR_SERVER:
DebugLog("NET (GetAction): Server error: Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_ERROR
Default:
DebugLog("NET (GetAction): Unclassified Server error (Assume Failure): Response code: "+Str(ns_ServerResponseCode))
Return NET_RESPONSE_ACTION_ERROR
End Select
End Function

;JUST A WRAPPER TO AVOID INCREDIBLY LONG LINE OF CODE. THIS MUST BE ALL ON ONE LINE
Function NET_BuildHandshakeString$(s1$,s2$,s3$,s4$,sProtocol$=NET_PROTOCOL_HANDLE_HTTP11)
If ((s2<>"") And (s2<>"/"))
s1$=FixPath(s1,True,True)
End If
Local Send$=Replace(NET_LOCAL_SEND_GET$+" "+s1$+s2$+" "+sProtocol$+CARRIAGE_RETURN$+"Host: "+s3$+CARRIAGE_RETURN$+"User-Agent: "+s4$+CARRIAGE_RETURN$,Chr(0),"")

Return Send
End Function

;RETURNS URL PROTOCOL PREFIX
Function NET_GetProtocolPrefix$(snt_Path$)
Local ns_Inst=Instr(snt_Path$,NET_PROTOCOL_SUFFIX)-1
If (ns_Inst)
Return Left(snt_Path$,ns_Inst+Len(NET_PROTOCOL_SUFFIX))
End If

Return ""
End Function

;RETURN IDENTIFIED PORT
Function NET_GetProtocolPort(snt_Path$)
Local ns_Port=NET_PORT_DEFAULT_HTTP_PORT

Local s_Protocol$=NET_GetProtocolPrefix(snt_Path)
If (s_Protocol<>"")

s_Protocol=Left(s_Protocol,Instr(s_Protocol,NET_PROTOCOL_SUFFIX)-1)
If (Lower(s_Protocol)=Lower(NET_PROTOCOL_HTTPS))
ns_Port=NET_PORT_DEFAULT_HTTPS_PORT
End If
End If
Return ns_Port
End Function

;RETURNS DOMAIN PATH WITHOUT HTTP PROTOCOL
Function NET_TrimProtocolHead$(snt_Path$)
Local snt_Trim$=NET_GetProtocolPrefix(snt_Path)
Local snt_Sector$=Right(snt_Path,Len(snt_Path)-Len(snt_Trim))

;Trim www. where applicable
; If (Left(Lower(snt_Sector),4)=(NET_WORLDWIDEWEB+"."))
; snt_Sector=Right(snt_Path,Len(snt_Path)-Len(snt_Trim)-4)
; End If

Return snt_Sector
End Function

; THIS IS THE MAIN DOWNLOAD FUNCTION AND CONTAINS DOWNLOADING STREAM TO FILE WRITING
Function NET_DownloadFile(snt_FullServerPath$, savepath$ = "", savefile$ = "",s_Protocol$=NET_PROTOCOL_HANDLE_HTTP11, nb_IgnoreNullFilename=False, Agent$=NET_USERAGENT)
;Strip protocol and return false if not "http"
Local sdr_DownloadPath$=FixPath(FixPath(savepath,True,False)+savefile$,False)

;Seperate Sector Path and FileName from original Server Path
;Seperate path and file from the link
Local nIterBytes
Local snt_ServerSector$=""
Local snt_ServerFile$=""

For nIterBytes = Len(snt_FullServerPath) To 1 Step -1
If (Mid(snt_FullServerPath,nIterBytes, 1) = "/")
snt_ServerSector$ = Trim(Left(snt_FullServerPath,nIterBytes))
snt_ServerFile$ = Right(snt_FullServerPath$, Len(snt_FullServerPath) - nIterBytes)
Exit
EndIf
Next

If ((snt_ServerFile = "") And (nb_IgnoreNullFilename=False))
;NULL FILENAME
DebugLog("NET (Download): Filename is null and no permission granted for null Filenames. Abort streaming attempt")
Return False
End If

Local snt_Domain$=NET_RetrieveDomain(snt_FullServerPath)

Local hst_TCPStream=NET_CreateTCPConnection(snt_FullServerPath)

If (Not(hst_TCPStream))
; NO TCP CONECTION
DebugLog("NET (Download): Connection could not be Established")
Return False
End If

DebugLog("NET (Download): Connection Established 0x"+Str(hst_TCPStream))

Local s_Send$=NET_BuildHandshakeString$(snt_ServerSector,snt_ServerFile,snt_Domain$,Agent$,s_Protocol$)

DebugLog("NET (Download): Attempting Handshake Negotiation")
DebugLog("NET (Download): "+Replace(s_Send,CARRIAGE_RETURN,"|"))

WriteLine hst_TCPStream,s_Send


Local ns_Reply=NET_ResolveServerResponse(hst_TCPStream)
Local nb_Action=NET_GetActionByServerResponse(ns_Reply)
Local nb_Success

Select (nb_Action)

Case NET_RESPONSE_ACTION_DOWNLOAD:
DebugLog("NET (Download): Continue to read from Stream..")
nb_Success=NET_DownloadFromStream(hst_TCPStream,sdr_DownloadPath)

Case NET_RESPONSE_ACTION_REDIRECT:
DebugLog("NET (Download): Stream authorised, but content relocated. Seeking Location: header")

Local NewPath$=NET_LocateRedirectionPathFromStream(hst_TCPStream)

If (NewPath<>"")
NewPath=FixPath(NewPath,False,True)
If (NewPath=snt_FullServerPath)
DebugLog("NET (Download): Redirect destination matches initial content target. Aborting to prevent infinite loop")
Else
DebugLog("NET (Download): Attempting to connect to redirect destination: "+NewPath)
Return NET_DownloadFile(NewPath,savepath,savefile,s_Protocol$,nb_IgnoreNullFilename,Agent)
End If
Else
DebugLog("NET (Download): Content redirection path is Null. No valid location")
End If

Default:
DebugLog("NET (Download): Unable to continue.")
NET_CloseTCPConnection(hst_TCPStream)
End Select

If (nb_Success)
If (FileType(sdr_DownloadPath)=1)
If (FileSize(sdr_DownloadPath))
DebugLog("NET (Download): Stream Read completed: "+FileSize(sdr_DownloadPath)+" bytes")
Else
DebugLog("NET (Download): Stream Data empty")
DeleteFile sdr_DownloadPath
nb_Success=False
End If
Else
DebugLog("NET (Download): Stream Read Was successful, but local Data could not be written")
;DOWNLOAD HAS FAILED
DeleteFile sdr_DownloadPath
nb_Success=False
End If
Else
DebugLog("NET (Download): File Stream Read Failed")
; DOWNLOAD HAD FAILED
DeleteFile sdr_DownloadPath
End If

Return nb_Success
End Function

;~IDEal Editor Parameters:
;~C#Blitz+
The "BlitzGET" and Download examples in the code archives, do not seem to be compatible with BlitzMax NG and my attempts to learn more from https://blitzmax.org/docs/en/tutorials/tcp_tutorial/ hit a bump because it always returns "BAD REQUEST"

Const pref1:String="https"
Const pref2:String="://"
Const domain:String="www.syntaxbomb.com"

Local hints:TAddrInfo = New TAddrInfo(AF_INET_, SOCK_STREAM_)'AF_INET_ = IPv4  -- or AF_INET6_ = IPv6 -- alternatively, use AF_UNSPEC_ for either protocol ---------------------- SOCK_STREAM = SOCK_DGRAM_ for UDP

Local socket:TSocket = TSocket.Create(hints)


If Not socket Then
    Print "Unable to create socket."
    End
End If

Print "Socket Created"

Local host:String = domain
Local port:Int = 80
Local infos:TAddrInfo[] = AddrInfo(host, port, hints)


If Not infos Then
    Print "Hostname could not be resolved."
    End
End If

' we'll try the first one
Local info:TAddrInfo = infos[0]

Print "IP address of " + host + " is " + info.HostIp()

'Connect() method returns true if successful

If Not socket.Connect(info) Then
    Print "Error connecting to remote"
    End
End If

Print "Socket connected to " + host + " on ip " + socket.RemoteIp()


' send some data to the remote server
Local message:String = "GET / HTTP/1.1~r~n~r~n"

If Not socket.Send(message, Size_T(message.length), 0) Then
    Print "Send failed."
    End
End If

Print "Send successful"
' now receive some data
Local buf:Byte Ptr[1024]
Local dataLen:Size_T = socket.Recv(buf, 1024)

If Not dataLen Then
    Print "No data"
    End
End If

Print String.FromUTF8String(buf)
socket.close

Example:

QuoteSocket Created
IP address of www.syntaxbomb.com is 213.175.217.229
Socket connected to www.syntaxbomb.com on ip 213.175.217.229
Send successful
HTTP/1.1 400 Bad Request
Date: Thu, 02 May 2024 09:30:12 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Connection: close
Content-Type: text/html

_________________


Is there an example of a means to send http request and download the html content at a given url ?

I guess you need less code to download from the internet in BlitzMax

SuperStrict
Global PlainText:String = LoadText("http::www.midiox.com")
Print PlainText
This is a technical minimum, but works for contact with not https-Servers like "http://www...."

This is the same for streams:

Const Header:String = "http::"
Const Adress:String = "www.anywhere.com/anypage.html"

FetchPage(Header + Adress)

Function FetchPage(Adress:String)
    ' fetches a complete html-page in steps:
    Local Whole:String
    Print Adress
    Local Http:TStream=ReadStream(Adress)
    While Eof(Http)=False
        Whole = Whole + ReadLine(Http)
    Wend
    CloseStream Http
    Print Whole
End Function

For the meanwhile mostly used https protocol, you have to do more. You need the LibCurl Library. Have a look on my tutorial here:

https://www.syntaxbomb.com/worklogs/fetch-a-html-document-from-a-https-server-with-blitzmax-ng/msg347059214/#msg347059214

You need to read the step 2 and step 4

Yep, I think I will need the secure http versions...

Thanks, Midimaster - I'll have a look over that LibCurl stuff. It seems to be exactly what I was looking for!