Pulling data from an API Endpoint into a CSV file

Right now I'm building datasets for some ML stuff by pulling data and storing in a csv.

Author

Carson Szeder
February 12, 2024

I figured it would probably be useful to share how to do this, It’s very simple, it’s only a handful of lines of code in addition to an api call.

import requests import pandas url = "https://query2.finance.yahoo.com/v1/finance/trending/US?count=500&useQuotes=true&fields=logoUrl%2CregularMarketChangePercent" headers = { "sec-ch-ua": '"Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"', 'referer':'https://finance.yahoo.com/rates/', "sec-ch-ua-mobile": "?0", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36", "sec-ch-ua-platform": '"Windows"', } response = requests.get(url, headers=headers) if response.status_code == 200: print(response.json()) else: print(f"Failed to fetch data: {response.status_code}")

This is the original code I used in a past video to pull data from a random API endpoint I found (Yahoo Trading stuff).

This prints trading data. Now, if we add this code:

jsonData = response.json() data = pandas.json_normalize(jsonData['finance']['result'][0]['quotes']) data.to_csv('finance.csv', index=False) else:

After the response, it’ll write the data inside of pandas.json_normalize to a CSV file.

That’s all! relatively straight forward process.