python3으로 requests get 호출(특정 url 컨텐츠 호출)

• 특정 URL의 컨텐츠를 호출하는 파이썬 스크립트
• 도메인 제외되고 URL의 Path만 있는 파일을 읽어 컨텐츠 호출

• python3 requests get 호출 스크립트

  $ vi request_get.py
  #! /bin/python3
  import sys
  import requests
  import time
  import logging
  
  logging.basicConfig(filename='./preload.log', level=logging.INFO)
  
  def make_server_list(file):
      serverList = []
      with open(file, mode='r') as serverfile:
          for server in serverfile:
              server = server.strip('\n')
              if not server:
                  break
              serverList.append(server)
      return serverList
  
  def curl_request(server, domain, path):
      headers = {
          "host": domain,
      }
      response = requests.get("http://"+server+path, headers=headers, timeout=10)
      #response = requests.head("http://"+server+path, headers=headers, timeout=10)
      return response
  
  if __name__ == '__main__':
      domain = sys.argv[1]
      urlArgs = sys.argv[2]
      serverArgs = sys.argv[3]
  
      if len(sys.argv) != 4:
          print("Insufficient arguments")
          sys.exit()
  
      with open(urlArgs, mode='r' ) as urlfile:
          serverList = make_server_list(serverArgs)
  
          for fileLine in urlfile:
              fileLine = fileLine.strip('\n')
              #scheme=fileLine.split('://',maxsplit=1)[0]
              #domain=fileLine.split('/',maxsplit=3)[2]
              path=fileLine
              for server in serverList:
                  start = time.perf_counter()
                  response = curl_request(server, domain, path)
                  request_time = time.perf_counter() - start
                  if response.status_code == 200:
                      print("server:" + server + "\t\tpreload url : " + domain + path + "\ttime : " + str(round(request_time,3))+"s")
                      logging.info("success preload : " + server + "\tpreload url : " + domain + path + "\ttime : " + str(round(request_time,3))+"s")
                  else:
                      print("error preload : " + server + "\tpreload url : " + domain + path)
                      logging.info("error preload : " + server + "\tpreload url : " + domain + path)
              #time.sleep(1)

  
  

• url 중에 도메인 제외한 리스트

  $ cat urllist.txt
  /test.txt

  
  

• 서버 리스트

  $ cat serverlist.txt 
  8.8.8.8

  
  

• 실행 예제 (실행 형식만 설명하지 실제 동작 X)

  $ python3 request_get.py test.co.kr urllist.txt serverlist.txt
  server:8.8.8.8            preload url : test.co.kr/test.txt       time : 3.037s