Networking - HTTPGet

_PJ_ · May 02, 2024, 10:55:54

I have been really struggling with what I expected to be a straightforward problem :
Downloading of html from a given URL

I had been using the following (( some dependencies for constants and filepath handling etc. are not included )) in B3D / B+ but was unabble to convert to BMax because of how BMax uses the Sockets etc.

Code: BASIC

;These provide a set of control codes. Typically the entry would be akin to &#x22; or &#x26 etc. 
Const HTM_CODE_LOWER_PREFIX$="&#x"
Const HTM_CODE_LOWER_SUFFIX$=";"

Const HTM_CARRIAGE_RETURN$="<br>"

Const HTM_STANDARDSYMBOLS_PREFIX$="<"
Const HTM_STANDARDSYMBOLS_SUFFIX$=">"

Const HTM_ENCODED_URL_CHARACTERS$="!£$^*()-=[]{}¬`;:'@#~|<>,?"

;	FIXES CONTROL CODES USED BY WEBPAGE HTML
Function HTM_CleanControlCodesText$(sText$)
	DebugLog("HTM (Clean HTM): Cleaning HTML Control Codes")
	
	; Non-Standard symbolisations
	Local IterFixes
	
	Local SeekEnd=0
	Local SeekStart=0
	Local SeekWidth=0
	
	Local MidString$=""
	
	; Fix for typical html control codes with <> tags
	SeekStart=Instr(sText,HTM_STANDARDSYMBOLS_PREFIX)
	If (SeekStart)
		While (SeekStart)
			SeekEnd=Instr(sText,HTM_STANDARDSYMBOLS_SUFFIX,SeekStart)
			SeekWidth=(SeekEnd-SeekStart)
			
			If (SeekStart*SeekEnd)
				MidString=Mid(sText,SeekStart,SeekWidth+1)
				DebugLog("HTM (Clean HTM): Removed instances of code: "+Chr(34)+MidString+Chr(34))
				sText=Replace(sText,MidString,"")
			End If
			SeekStart=Instr(sText,HTM_STANDARDSYMBOLS_PREFIX)
		Wend	
	End If	
	
	; FIX FOR ATYPICAL CONTROL CODES FOR SYMBOLS WITH CONVERTED HEX NOTATIONS (i.e. &#xnn )
	SeekStart=Instr(Lower(sText),HTM_CODE_LOWER_PREFIX)
	If (SeekStart)
		While (SeekStart)
			SeekEnd=Instr(Lower(sText),HTM_CODE_LOWER_SUFFIX,SeekStart)
			SeekWidth=(SeekEnd-SeekStart)
			If (SeekStart*SeekEnd)
				MidString=Mid(sText,SeekStart,SeekWidth+1)
				DebugLog("HTM (Clean HTM): Removed instances of code: "+Chr(34)+MidString+Chr(34))
				sText=Replace(sText,MidString,"")
			End If
			SeekStart=Instr(Lower(sText),HTM_CODE_LOWER_PREFIX)
		Wend	
	End If	
	
	; Remove any remaining ascii control codes
	DebugLog("HTM (Clean HTM): Removing ASCII code instances")
	Local IterAsc
	For IterAsc=0 To 31
		sText=Replace(sText,Chr(IterAsc),"")
	Next
	
	Return sText$
End Function

;ENCODES CHARACTERS IN A STRING SUITABLE FOR URL ENCODING
Function HTM_EncodeURL$(sText$)	
	
	DebugLog("HTM (Fix url): Encoding "+Chr(34)+sText+Chr(34))
	
	sText=Replace(Lower(Trim(sText)),"_","+")
	sText=Replace(Lower(Trim(sText))," ","+")
	sText=Replace(sText,"&","+")
	sText=Replace(sText,STRING_PERCENT,"percent")
	sText=Replace(sText,Chr(34),"")
	
	Local Count
	Local Char$
	For Count=1 To Len(sText)
		Char=(Mid(sText,Count,1))
		Replace(sText,Char,HTM_URLEncode(Char))
	Next
	
	;Deal with multiple +'s
	While Instr(sText,"++")
		sText=(Replace(sText,"++","+"))
	Wend
	
	sText=Lower(Trim(sText))
	
	DebugLog("HTM (Fix url): Result of encoding: "+Chr(34)+sText+Chr(34))
	
	Return sText
	
End Function

;ENCODES NON-ALPHANUMERIC CHARACTERS WITH VALID URL HTML CODES
Function HTM_URLEncode(s_Char)
	If (Not(GetIsAlphanumericCharacter(s_Char)))
		If (Instr(HTM_ENCODED_URL_CHARACTERS,s_Char))
			s_Char=STRING_PERCENT+Str(Asc(s_Char))
		End If
	End If
	Return s_Char
End Function
;~IDEal Editor Parameters:
;~C#Blitz+

;________________________________________________________________________________________________________________

Const NET_PROTOCOL_HTTP$="http"
Const NET_PROTOCOL_HTTPS$="https"
Const NET_PROTOCOL_FTP$="ftp"
Const NET_WORLDWIDEWEB$="www"

Const NET_PROTOCOL_HANDLE_HTTP09$="HTTP/0.9"
Const NET_PROTOCOL_HANDLE_HTTP10$="HTTP/1.0"
Const NET_PROTOCOL_HANDLE_HTTP11$="HTTP/1.1"

Const NET_VALID_HEX$="123456789ABCDEF"

Const NET_USERAGENT$="EKD_Engine_Agent"
Const NET_BROWSERAGENT_IE11$="Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko"
Const NET_BROWSERAGENT_IE8$="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2)"
Const NET_BROWSERAGENT_IE6$="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"

Const NET_PROTOCOL_SUFFIX$="://";cs_STRING_COLON+cs_STRING_WEB_SEPARATOR+cs_STRING_WEB_SEPARATOR

Const NET_DLE_FLAGS=7

Const NET_DLE_COM=1
Const NET_DLE_UK=2
Const NET_DLE_ORG=4

Const NET_DOMAIN_COM$="com"
Const NET_DOMAIN_UK$="co.uk"
Const NET_DOMAIN_ORG$="org"

Const NET_LOCAL_SEND_GET$="GET"
Const NET_LOCAL_SEND_POST$="POST"

Const NET_INVALIDFILECHARS$="=*:?<>|"

Const NET_SERVER_MESSAGE_REDIRECT$="location"
Const NET_SERVER_MESSAGE_LENGTH$="content-length"
Const NET_SERVER_MESSAGE_ENCODING$="transfer-encoding"
Const NET_SERVER_MESSAGE_CHUNKED$="chunked"

Const NET_PORT_DEFAULT_HTTP_PORT=80
Const NET_PORT_DEFAULT_HTTPS_PORT=443

Const NET_RESPONSE_OK=200
Const NET_RESPONSE_OK_CREATE=201
Const NET_RESPONSE_OK_ACCEPT=202
Const NET_RESPONSE_OK_PARTIAL=203
Const NET_RESPONSE_OK_NOREPLY=204

Const NET_RESPONSE_OK_MESSAGE=220

Const NET_RESPONSE_BAD_REQUEST=400
Const NET_RESPONSE_BAD_UNAUTHORISED=401
Const NET_RESPONSE_BAD_PAYMENT=402
Const NET_RESPONSE_BAD_FORBIDDEN=403
Const NET_RESPONSE_BAD_NOTFOUND=404

Const NET_RESPONSE_ERR_SERVER=500
Const NET_RESPONSE_ERR_NOTIMPLEMENTED=501
Const NET_RESPONSE_ERR_BANDWIDTH=502
Const NET_RESPONSE_ERR_TIMEOUT=503

Const NET_RESPONSE_REDIR_MOVED=301
Const NET_RESPONSE_REDIR_FOUND=302
Const NET_RESPONSE_REDIR_METHOD=303
Const NET_RESPONSE_REDIR_NOCHANGE=304

Const NET_RESPONSE_QUERY_PASS=331

Const NET_RESPONSE_ACTION_ERROR=0
Const NET_RESPONSE_ACTION_DOWNLOAD=1
Const NET_RESPONSE_ACTION_REDIRECT=2

Const NET_CHUNKSIZE=4096

Const NET_TIMEOUT_MILLISECS=5000

 ; WRITES BYTES FROM STREAM TO FILE
Function NET_DownloadFromStream(hst_Stream, DownloadPath$)
	Local n4_ContentLength=-1
	Local nb_Chunked = False
	
	Local s_Line$
	
	Local ns_Pos
	Local s_Part1$
	Local s_Part2$
	
	;Recurse Bytes from stream...
	Repeat
		
		s_Line$ = Trim(NET_ReadStreamLine$(hst_Stream))
		
		DebugLog("NET (DownloadFromStream): Server: "+s_Line)
		
		If (s_Line$ = "")
			;END OF STREAM!
			DebugLog("NET (DownloadFromStream): Null data line from server")
			Exit
		End If
		
		ns_Pos = Instr(s_Line,":")
		s_Part1 = Trim(Left(s_Line, ns_Pos - 1))
		s_Part2 = Trim(Right(s_Line, Len(s_Line) - ns_Pos))
		
		Select (Lower(s_Part1$))
			Case NET_SERVER_MESSAGE_LENGTH:
				n4_ContentLength = Int(s_Part2)
			Case NET_SERVER_MESSAGE_ENCODING:
				If (Lower(s_Part2$) = Lower(NET_SERVER_MESSAGE_CHUNKED))
					nb_Chunked = True
				End If	
		End Select
		
	Until (s_Line="")
	
	If (Eof(hst_Stream)<0)
		DebugLog("NET (DownloadFromStream): Error in processing stream. Aborting process")
		NET_CloseTCPConnection(hst_Stream)
		Return False
	End If
	
	Local hst_File=False
	
	hst_File = WriteFile(DownloadPath$)
	
	Local mb_Bank
	Local n4_Available
	
	If (Not(n4_ContentLength))
		DebugLog("NET (DownloadFromStream): Data length not provided")
		
		CloseFile hst_File
		hst_File=False
		
		NET_CloseTCPConnection(hst_Stream)
		
		Return (FileType(DownloadPath)=1)
		
	Else 
		
		DebugLog("NET (DownloadFromStream): Retreiving data form server...")
		
		If (Not(nb_Chunked))
			
			DebugLog("NET (DownloadFromStream): Data not served in chunked format")
			
			mb_Bank = CreateBank(NET_CHUNKSIZE)
			ns_Pos = 0
			
			Repeat
				n4_Available = (n4_ContentLength - ns_Pos)
				
				If (n4_Available > NET_CHUNKSIZE)
					ReadBytes mb_Bank, hst_Stream, 0, NET_CHUNKSIZE
					WriteBytes mb_Bank, hst_File, 0, NET_CHUNKSIZE
					ns_Pos = ns_Pos + NET_CHUNKSIZE
					
				Else
					
					ReadBytes mb_Bank, hst_Stream, 0, n4_Available
					WriteBytes mb_Bank, hst_File, 0, n4_Available
					
					ns_Pos=n4_Available
					Exit
				EndIf
				
			Until ns_Pos=0
			
			FreeBank mb_Bank
			
			CloseFile hst_File
			NET_CloseTCPConnection(hst_Stream)
			hst_File=False
			
			Return (FileType(DownloadPath)=1)
		End If
		
		If (nb_Chunked)
			DebugLog("NET (DownloadFromStream): Data served in chunked format")
			
			If (Not(hst_File))
				hst_File =WriteFile(DownloadPath)
			End If
			
			mb_Bank = CreateBank(NET_CHUNKSIZE)
			Local ns_IterBits
			
			Repeat
				
				s_Line$ = Trim(Upper(NET_ReadStreamLine$(hst_Stream)))
				
				ns_Pos = 0
				
				For ns_IterBits = 1 To Len(s_Line$)
					ns_Pos = 16 * ns_Pos + Instr(NET_VALID_HEX,Mid$(s_Line, ns_IterBits, 1))
				Next
				
				If (Not(ns_Pos))
					Exit
				End If
				
				If (BankSize(mb_Bank) < ns_Pos)
					ResizeBank mb_Bank, ns_Pos
				End If	
				
				ReadBytes mb_Bank, hst_Stream, 0, ns_Pos
				WriteBytes mb_Bank, hst_File, 0, ns_Pos
				
				ReadShort(hst_Stream)
				
			Until ns_Pos=0
			
			FreeBank mb_Bank
			
			CloseFile hst_File
			hst_File=False
			
			NET_CloseTCPConnection(hst_Stream)
			Return (FileType(DownloadPath)=1)
		Else
			DebugLog("NET (DownloadFromStream): Data served in invalid format")
			
			NET_CloseTCPConnection(hst_Stream)
			
			CloseFile hst_File
			hst_File=False
			
			Return False
		EndIf
	End If
End Function

;Function NET_ReadStreamBytes(hst_StreamHandle,hmb_BankHandle,n4_ByteCount=1,n4_Offset=0)
;	If (Not(hst_StreamHandle))
;		DebugLog("NET (ReadBytes: Stream Handle is null")
;		Return
;	End If
;	
;	WriteBytes(hmb_BankHandle,hst_StreamHandle,n4_Offset,n4_ByteCount)
;	
;End Function

;RETURNS LINE STRING READ FROM STREAM 
Function NET_ReadStreamLine$(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadLinet): Stream Handle is null")
		Return ""
	End If	
	
	Local s_Line$=ReadLine(hst_StreamHandle)
	Return s_Line$
End Function

;RETURNS STRING READ FROM STREAM 
Function NET_ReadStreamString$(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadString): Stream Handle is null")
		Return ""
	End If	
	
	Local s_String$=ReadString(hst_StreamHandle)
	
	Return s_String
	
End Function	

;RETURNS 4-BYTE INTEGER READ FROM STREAM 
Function NET_ReadStreamInt%(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadInt): Stream Handle is null")
		Return 0
	End If	
	
	Local n4_Int$=ReadInt(hst_StreamHandle)
	
	Return n4_Int
	
End Function	

;RETURNS 2-BYTE INTEGER READ FROM STREAM 
Function NET_ReadStreamShort(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadShort): Stream Handle is null")
		Return 0
	End If	
	
	Local ns_Short=ReadShort(hst_StreamHandle)
	
	Return ns_Short
	
End Function		

;RETURNS 1-BYTE INTEGER READ FROM STREAM 
Function NET_ReadStreamByte(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadByte): Stream Handle is null")
		Return 0
	End If	
	Local m_Byte=ReadByte(hst_StreamHandle)
	
	Return m_Byte
End Function		

;RETURNS 4-BYTE FLOATING POINT VALUE READ FROM STREAM 
Function NET_ReadStreamFloat#(hst_StreamHandle)
	If (Not(hst_StreamHandle))
		DebugLog("NET (ReadFloat): Stream Handle is null")
		Return 0.0
	End If	
	
	Local f_Float#=ReadFloat(hst_StreamHandle)
	
	Return f_Float#
	
End Function

; SIMPLIFIES HTTP SERVER RESPONSE
Function NET_ResolveServerResponse(hst_StreamHandle)
	
	If (Not(hst_StreamHandle))
		DebugLog("NET (Resolve Response): Null stream handle, response resolved as bad")
		Return NET_RESPONSE_BAD_REQUEST
	End If
	
	Local sLine$ = ReadLine(hst_StreamHandle)
	
	DebugLog("NET (Resolve Response): Response "+Chr(34)+sLine+Chr(34))
	
	If (sLine="")
		DebugLog("NET (Resolve Response): No response message from server")
		Return NET_RESPONSE_BAD_REQUEST
	End If	
	
	Local ns_Pos1 = Instr(sLine$," ")
	Local ns_Pos2 = Instr(sLine$," ", ns_Pos1 + 1)
	
	If (Not(ns_Pos1*ns_Pos2))
		DebugLog("NET (Resolve Response): Incorrect message format from server")
		Return NET_RESPONSE_BAD_REQUEST
	End If	
	
	Return Int(Mid(sLine$,ns_Pos1, ns_Pos2 - ns_Pos1))
End Function	

; WHEN ENCOUNTERING 3XX RESPONSE FROM SERVER (CONTENT REDIRECTION), NEXT SERVER TRANSMISSION SHOULD BE REDIRECTION PATH
Function NET_LocateRedirectionPathFromStream$(hst_Stream)
	Local s_Line$
	
	Local Redir$
	Local Pos
	
	Local snt_NewPath$
	
	Repeat
		s_Line$ = Trim(ReadLine(hst_Stream))
		
		DebugLog("NET (Redirect): Server: "+s_Line)
		
		If (s_Line="")
			Exit
		End If	
		
		Pos = Instr(s_Line$,":")
		
		If (Pos)
			If (Lower(Trim(Left(s_Line$,Pos-1)))=Lower(NET_SERVER_MESSAGE_REDIRECT))
				snt_NewPath = FixPath(Trim(Right(s_Line$, Len(s_Line$) - Pos)),False,True)
				DebugLog("NET (Redirect): Obtained redirected path for content: "+snt_NewPath)
				DebugLog("NET (Redirect): Closing current stream to commence new download from relocation.")
				NET_CloseTCPConnection(hst_Stream)
					
				Return snt_NewPath$
			End If
		End If
		
	Until s_Line=""
	
	DebugLog("NET (Redirect): Unable to obtain redirection path. Download aborted")
	NET_CloseTCPConnection(hst_Stream)
	
	Return ""
End Function	

; RETURNS DOMAIN NAME AND DLE ONLY
Function NET_RetrieveDomain$(snt_Path$)
	Local s_Sectors$=NET_TrimProtocolHead(snt_Path)
	snt_Path$ = Left(s_Sectors,Instr(s_Sectors,"/")-1)
	
	Return snt_Path
End Function

; OPENS TCP CONNECTION STREAM TO DOWNLOAD PATH AND RETURNS HANDLE
Function NET_CreateTCPConnection(snt_FullServerPath$)
	snt_FullServerPath=FixPath(snt_FullServerPath,False,True)
	
	Local s_Protocol$=NET_GetProtocolPrefix$(snt_FullServerPath$)
	Local ns_Port=NET_GetProtocolPort(snt_FullServerPath)
	Local snt_Domain$=NET_RetrieveDomain(snt_FullServerPath)
	DebugLog("NET (Create TCP): Connection Polling "+snt_Domain+" via port "+Str(ns_Port))	
	;Open TCP stream
	Local hst_tcp = OpenTCPStream(snt_Domain$, ns_Port )
	DebugLog("NET (Create TCP): Connection 0x"+Str(hst_tcp)+" opened to "+DottedIP(TCPStreamIP(hst_tcp))+":"+Str(ns_Port))
	Return hst_tcp
End Function

; CLOSES TCP STREAM CONNECTION
Function NET_CloseTCPConnection(hst_TCP=False)
	If (hst_TCP)
		DebugLog("NET (Close TCP): Stream 0x"+Str(hst_TCP)+" to "+DottedIP(TCPStreamIP(hst_TCP))+":"+Str(TCPStreamPort( hst_TCP ))+" closed")
		CloseTCPStream(hst_TCP)
	Else
		DebugLog("NET (Close TCP): Attempted To close Null stream")
	End If
End Function

;RETUIRNS A SIMPLIFIED VALUE DETERMINING APPROPRIATE ACTION ACCORDING TO RESPONSE VALUE
Function NET_GetActionByServerResponse(ns_ServerResponseCode=0)
	Select (ns_ServerResponseCode)
		Case NET_RESPONSE_OK, NET_RESPONSE_OK_ACCEPT, NET_RESPONSE_OK_CREATE, NET_RESPONSE_OK_MESSAGE, NET_RESPONSE_OK_NOREPLY, NET_RESPONSE_OK_PARTIAL:
			DebugLog("NET (GetAction): Permission granted: Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_DOWNLOAD
		Case NET_RESPONSE_REDIR_MOVED,NET_RESPONSE_REDIR_FOUND,NET_RESPONSE_REDIR_METHOD, NET_RESPONSE_REDIR_NOCHANGE:
			DebugLog("NET (GetAction): Destination redirected: Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_REDIRECT
		Case NET_RESPONSE_BAD_NOTFOUND:
			DebugLog("NET (GetAction): Destination not found : "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_ERROR
		Case NET_RESPONSE_BAD_FORBIDDEN,NET_RESPONSE_BAD_REQUEST,NET_RESPONSE_BAD_PAYMENT,NET_RESPONSE_BAD_UNAUTHORISED:
			DebugLog("NET (GetAction): Permission refused: Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_ERROR
		Case NET_RESPONSE_ERR_TIMEOUT:
			DebugLog("NET (GetAction): Attempt timed-out: Response code: "+Str(ns_ServerResponseCode))
		Case NET_RESPONSE_ERR_BANDWIDTH:
			DebugLog("NET (GetAction): Server busy: Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_ERROR			
		Case NET_RESPONSE_ERR_NOTIMPLEMENTED,NET_RESPONSE_ERR_SERVER:
			DebugLog("NET (GetAction): Server error: Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_ERROR
		Default:
			DebugLog("NET (GetAction): Unclassified Server error (Assume Failure): Response code: "+Str(ns_ServerResponseCode))
			Return NET_RESPONSE_ACTION_ERROR
	End Select
End Function	

;JUST A WRAPPER TO AVOID INCREDIBLY LONG LINE OF CODE. THIS MUST BE ALL ON ONE LINE
Function NET_BuildHandshakeString$(s1$,s2$,s3$,s4$,sProtocol$=NET_PROTOCOL_HANDLE_HTTP11)
	If ((s2<>"") And (s2<>"/"))
		s1$=FixPath(s1,True,True)
	End If
	Local Send$=Replace(NET_LOCAL_SEND_GET$+" "+s1$+s2$+" "+sProtocol$+CARRIAGE_RETURN$+"Host: "+s3$+CARRIAGE_RETURN$+"User-Agent: "+s4$+CARRIAGE_RETURN$,Chr(0),"")
	
	Return Send
End Function

;RETURNS URL PROTOCOL PREFIX 
Function NET_GetProtocolPrefix$(snt_Path$)
	Local ns_Inst=Instr(snt_Path$,NET_PROTOCOL_SUFFIX)-1
	If (ns_Inst)
		Return Left(snt_Path$,ns_Inst+Len(NET_PROTOCOL_SUFFIX))
	End If
	
	Return ""
End Function

	;RETURN IDENTIFIED PORT
Function NET_GetProtocolPort(snt_Path$)
	Local ns_Port=NET_PORT_DEFAULT_HTTP_PORT
	
	Local s_Protocol$=NET_GetProtocolPrefix(snt_Path)
	If (s_Protocol<>"")
		
		s_Protocol=Left(s_Protocol,Instr(s_Protocol,NET_PROTOCOL_SUFFIX)-1)
		If (Lower(s_Protocol)=Lower(NET_PROTOCOL_HTTPS))
			ns_Port=NET_PORT_DEFAULT_HTTPS_PORT
		End If
	End If	
	Return ns_Port
End Function

;RETURNS DOMAIN PATH WITHOUT HTTP PROTOCOL
Function NET_TrimProtocolHead$(snt_Path$)
	Local snt_Trim$=NET_GetProtocolPrefix(snt_Path)
	Local snt_Sector$=Right(snt_Path,Len(snt_Path)-Len(snt_Trim))
	
	;Trim www. where applicable
;	If (Left(Lower(snt_Sector),4)=(NET_WORLDWIDEWEB+"."))
;		snt_Sector=Right(snt_Path,Len(snt_Path)-Len(snt_Trim)-4)
;	End If
	
	Return snt_Sector
End Function

; THIS IS THE MAIN DOWNLOAD FUNCTION AND CONTAINS DOWNLOADING STREAM TO FILE WRITING
Function NET_DownloadFile(snt_FullServerPath$, savepath$ = "", savefile$ = "",s_Protocol$=NET_PROTOCOL_HANDLE_HTTP11, nb_IgnoreNullFilename=False, Agent$=NET_USERAGENT)
	;Strip protocol and return false if not "http"
	Local sdr_DownloadPath$=FixPath(FixPath(savepath,True,False)+savefile$,False)
	
	;Seperate Sector Path and FileName from original Server Path
	;Seperate path and file from the link
	Local nIterBytes
	Local snt_ServerSector$=""
	Local snt_ServerFile$=""
	
	For nIterBytes = Len(snt_FullServerPath) To 1 Step -1
		If (Mid(snt_FullServerPath,nIterBytes, 1) = "/")
			snt_ServerSector$ = Trim(Left(snt_FullServerPath,nIterBytes))
			snt_ServerFile$ = Right(snt_FullServerPath$, Len(snt_FullServerPath) - nIterBytes)
			Exit
		EndIf
	Next
	
	If ((snt_ServerFile = "") And (nb_IgnoreNullFilename=False))
		;NULL FILENAME
		DebugLog("NET (Download): Filename is null and no permission granted for null Filenames. Abort streaming attempt")
		Return False
	End If
	
	Local snt_Domain$=NET_RetrieveDomain(snt_FullServerPath)
	
	Local hst_TCPStream=NET_CreateTCPConnection(snt_FullServerPath)
	
	If (Not(hst_TCPStream))
		; NO TCP CONECTION
		DebugLog("NET (Download): Connection could not be Established")
		Return False
	End If
	
	DebugLog("NET (Download): Connection Established 0x"+Str(hst_TCPStream))	
	
	Local s_Send$=NET_BuildHandshakeString$(snt_ServerSector,snt_ServerFile,snt_Domain$,Agent$,s_Protocol$)
	
	DebugLog("NET (Download): Attempting Handshake Negotiation")
	DebugLog("NET (Download): "+Replace(s_Send,CARRIAGE_RETURN,"|"))
	
	WriteLine hst_TCPStream,s_Send
	
	
	Local ns_Reply=NET_ResolveServerResponse(hst_TCPStream)	
	Local nb_Action=NET_GetActionByServerResponse(ns_Reply)
	Local nb_Success
	
	Select (nb_Action)
			
		Case NET_RESPONSE_ACTION_DOWNLOAD:
			DebugLog("NET (Download): Continue to read from Stream..")
			nb_Success=NET_DownloadFromStream(hst_TCPStream,sdr_DownloadPath)
			
		Case NET_RESPONSE_ACTION_REDIRECT:
			DebugLog("NET (Download): Stream authorised, but content relocated. Seeking Location: header")
			
			Local NewPath$=NET_LocateRedirectionPathFromStream(hst_TCPStream)
			
			If (NewPath<>"")
				NewPath=FixPath(NewPath,False,True)
				If (NewPath=snt_FullServerPath)
					DebugLog("NET (Download): Redirect destination matches initial content target. Aborting to prevent infinite loop")	
				Else
					DebugLog("NET (Download): Attempting to connect to redirect destination: "+NewPath)	
					Return NET_DownloadFile(NewPath,savepath,savefile,s_Protocol$,nb_IgnoreNullFilename,Agent)
				End If
			Else
				DebugLog("NET (Download): Content redirection path is Null. No valid location")
			End If
			
		Default:
			DebugLog("NET (Download): Unable to continue.")
			NET_CloseTCPConnection(hst_TCPStream)
	End Select		
	
	If (nb_Success)
		If (FileType(sdr_DownloadPath)=1)
			If (FileSize(sdr_DownloadPath))
				DebugLog("NET (Download): Stream Read completed: "+FileSize(sdr_DownloadPath)+" bytes")
			Else
				DebugLog("NET (Download): Stream Data empty")
				DeleteFile sdr_DownloadPath
				nb_Success=False
			End If	
		Else
			DebugLog("NET (Download): Stream Read Was successful, but local Data could not be written")
			;DOWNLOAD HAS FAILED
			DeleteFile sdr_DownloadPath
			nb_Success=False
		End If
	Else
		DebugLog("NET (Download): File Stream Read Failed")
		; DOWNLOAD HAD FAILED
		DeleteFile sdr_DownloadPath
	End If	
	
	Return nb_Success
End Function 

;~IDEal Editor Parameters:
;~C#Blitz+

The "BlitzGET" and Download examples in the code archives, do not seem to be compatible with BlitzMax NG and my attempts to learn more from https://blitzmax.org/docs/en/tutorials/tcp_tutorial/ hit a bump because it always returns "BAD REQUEST"

Code: BASIC

Const pref1:String="https"
Const pref2:String="://"
Const domain:String="www.syntaxbomb.com"

Local hints:TAddrInfo = New TAddrInfo(AF_INET_, SOCK_STREAM_)'AF_INET_ = IPv4  -- or AF_INET6_ = IPv6 -- alternatively, use AF_UNSPEC_ for either protocol ---------------------- SOCK_STREAM = SOCK_DGRAM_ for UDP

Local socket:TSocket = TSocket.Create(hints)


If Not socket Then
    Print "Unable to create socket."
    End
End If

Print "Socket Created"

Local host:String = domain
Local port:Int = 80
Local infos:TAddrInfo[] = AddrInfo(host, port, hints)


If Not infos Then
    Print "Hostname could not be resolved."
    End
End If

' we'll try the first one
Local info:TAddrInfo = infos[0]

Print "IP address of " + host + " is " + info.HostIp()

'Connect() method returns true if successful

If Not socket.Connect(info) Then
    Print "Error connecting to remote"
    End
End If

Print "Socket connected to " + host + " on ip " + socket.RemoteIp()


' send some data to the remote server
Local message:String = "GET / HTTP/1.1~r~n~r~n"

If Not socket.Send(message, Size_T(message.length), 0) Then
    Print "Send failed."
    End
End If

Print "Send successful"
' now receive some data
Local buf:Byte Ptr[1024]
Local dataLen:Size_T = socket.Recv(buf, 1024)

If Not dataLen Then
    Print "No data"
    End
End If

Print String.FromUTF8String(buf)
socket.close

Example:

QuoteSocket Created
IP address of www.syntaxbomb.com is 213.175.217.229
Socket connected to www.syntaxbomb.com on ip 213.175.217.229
Send successful
HTTP/1.1 400 Bad Request
Date: Thu, 02 May 2024 09:30:12 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Connection: close
Content-Type: text/html

_________________

Is there an example of a means to send http request and download the html content at a given url ?

Midimaster · May 02, 2024, 11:58:32

I guess you need less code to download from the internet in BlitzMax

Code: BASIC

SuperStrict
Global PlainText:String = LoadText("http::www.midiox.com")
Print PlainText

This is a technical minimum, but works for contact with not https-Servers like "http://www...."

This is the same for streams:

Code: BASIC

Const Header:String = "http::"
Const Adress:String = "www.anywhere.com/anypage.html"

FetchPage(Header + Adress)

Function FetchPage(Adress:String)
    ' fetches a complete html-page in steps:
    Local Whole:String
    Print Adress
    Local Http:TStream=ReadStream(Adress)
    While Eof(Http)=False 
        Whole = Whole + ReadLine(Http)
    Wend 
    CloseStream Http
    Print Whole
End Function

For the meanwhile mostly used https protocol, you have to do more. You need the LibCurl Library. Have a look on my tutorial here:

https://www.syntaxbomb.com/worklogs/fetch-a-html-document-from-a-https-server-with-blitzmax-ng/msg347059214/#msg347059214

You need to read the step 2 and step 4

_PJ_ · May 02, 2024, 13:04:17

Yep, I think I will need the secure http versions...

Thanks, Midimaster - I'll have a look over that LibCurl stuff. It seems to be exactly what I was looking for!

SyntaxBomb - Indie Coders

Networking - HTTPGet

_PJ_

Midimaster

_PJ_