輕鬆抓取Twitter關注數據，使用Python與Selenium

抓取數據示例

{
  "userId": "95092020",
  "isBlueVerified": true,
  "following": false,
  "canDm": false,
  "canMediaTag": false,
  "createdAt": "Sun Dec 06 23:33:02 +0000 2009",
  "defaultProfile": false,
  "defaultProfileImage": false,
  "description": "Best-Selling Author | Clinical Psychologist | #1 Education Podcast | Enroll to @petersonacademy now:",
  "fastFollowersCount": 0,
  "favouritesCount": 161,
  "followersCount": 5613000,
  "friendCount": 1686,
  "hasCustomTimelines": true,
  "isTranslator": false,
  "listedCount": 14572,
  "location": "",
  "mediaCount": 7318,
  "name": "Dr Jordan B Peterson",
  "normalFollowersCount": 5613000,
  "pinnedTweetIdsStr": [
    "1849105729438790067"
  ],
  "possiblySensitive": false,
  "profileImageUrlHttps": "https://pbs.twimg.com/profile_images/1407056014776614923/TKBC60e1_normal.jpg",
  "profileInterstitialType": "",
  "username": "jordanbpeterson",
  "statusesCount": 51343,
  "translatorType": "none",
  "verified": false,
  "wantRetweets": false,
  "withheldInCountries": []
}

直接運行代碼，無需設置
我們的指南提供了完整的即用代碼，無縫抓取Twitter關注數據。使用Python和Selenium，自動化數據收集並高效捕獲性能日誌。無需額外設置，解鎖Twitter洞察！

步驟1：設置環境

首先，安裝Selenium進行瀏覽器自動化：

pip install -r requirements.txt

步驟2：下載ChromeDriver

下載ChromeDriver以便Selenium與Chrome瀏覽器互動。獲取鏈接：ChromeDriver下載

步驟3：設置Chrome選項

self.options = webdriver.ChromeOptions()
user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'
self.options.add_argument(f'user-agent={user_agent}')
self.options.add_argument('--disable-gpu')
self.options.add_argument('--no-sandbox')
self.options.add_argument('--disable-dev-shm-usage')
self.options.add_argument(f"--remote-debugging-port={remote_debugging_port}")

js_script_name = modify_random_canvas_js()
self.browser = self.get_browser(script_files=[js_script_name], record_network_log=True, headless=True)

步驟4：訪問目標頁面

self.browser.switch_to.new_window('tab')
url = 'https://x.com/1_usd_promotion/following'
self.browser.get(url=url)

time.sleep(2)

exist_entry_id = []

self.get_network(exist_entry_id, result_list)

print(f'推文結果長度 = {len(result_list)}')

步驟5：獲取瀏覽器性能日誌

performance_log = self.browser.get_log("performance")
for packet in performance_log:

    msg = packet.get("message")
    message = json.loads(packet.get("message")).get("message")
    packet_method = message.get("method")

    if "Network" in packet_method and 'Following' in msg:

        request_id = message.get("params").get("requestId")

        resp = self.browser.execute_cdp_cmd('Network.getResponseBody', {'requestId': request_id})

步驟6：從響應中提取數據

body = resp.get("body")
body = json.loads(body)
instructions = body['data']['user']['result']['timeline']['timeline'].get('instructions', None)
if not instructions:
    continue
for instruction in instructions:
    entries = instruction.get('entries', None)

步驟7：示例響應數據

{
  "userId": "95092020",
  "isBlueVerified": true,
  "following": false,
  "canDm": false,
  "canMediaTag": false,
  "createdAt": "Sun Dec 06 23:33:02 +0000 2009",
  "defaultProfile": false,
  "defaultProfileImage": false,
  "description": "Best-Selling Author | Clinical Psychologist | #1 Education Podcast | Enroll to @petersonacademy now:",
  "fastFollowersCount": 0,
  "favouritesCount": 161,
  "followersCount": 5613000,
  "friendCount": 1686,
  "hasCustomTimelines": true,
  "isTranslator": false,
  "listedCount": 14572,
  "location": "",
  "mediaCount": 7318,
  "name": "Dr Jordan B Peterson",
  "normalFollowersCount": 5613000,
  "pinnedTweetIdsStr": [
    "1849105729438790067"
  ],
  "possiblySensitive": false,
  "profileImageUrlHttps": "https://pbs.twimg.com/profile_images/1407056014776614923/TKBC60e1_normal.jpg",
  "profileInterstitialType": "",
  "username": "jordanbpeterson",
  "statusesCount": 51343,
  "translatorType": "none",
  "verified": false,
  "wantRetweets": false,
  "withheldInCountries": []
}

步驟8：重要注意事項

• 登錄Twitter並獲取你的Twitter Cookie。了解如何獲取Twitter Cookie
• 使用Apify的API
• 從GitHub獲取完整代碼
• 加入我們的討論群！點擊這裡