Python Tutorial: Working with JSON Data using the json Module
- March 24, 2024
- Posted by: MainInstructor
- Category: Go JavaScript Python
Video Title: Python Tutorial: Working with JSON Data using the json Module
Hey there how’s it going everybody and this video we’ll be learning how to work with JSON data in Python now if you’ve never used JSON basically it’s a very common data format for storing some information and you’re bound to run into this at some point if you haven’t already
So you’ll see JSON used a lot when fetching data from online api’s but it’s also used for configuration files and different kinds of data that can be saved on your local machine so JSON stands for JavaScript object notation but don’t get caught up on the JavaScript part of the name it was
Inspired by JavaScript but is now independent of any one language so pretty much every language these days has libraries for parsing and generating JSON data so let’s go ahead and get started and see what this looks like in Python so first of all i’ve imported the
Json library here at the top of the file and this is part of the standard library so there’s no need to install anything and I have a multi-line string here that is valid JSON and you can see that this almost looks like a Python dictionary so
This JSON here has a key called people and the value of people is an array of more objects so in this case it’s just two more objects and each object has a key of name a key of phone emails and has license which is just a boolean of
True or false so right now this is just a Python string that happens to be valid JSON so let’s see how to load this into a Python object so that we can work with the data more easily so to load this into Python from a string we can use the
Json dot load s method and so to do this we can say data equals json dot load s and we want to pass in that string into that load s function so i’ll save that and now let’s print this out so I will just print out data and run that ok so
We can see that it prints that out but it’s not very clean it’s kind of bunched together but it looks like a Python dictionary so if I check the type of this data variable and just print out the type of data save that and run it
Then we can see that that is a dictionary so when we load JSON into a Python object it uses the following conversion table let me pull this up here to convert this into a Python object so when it parses out that string if it finds a JSON object it converts
That to a Python dictionary it converts JSON arrays into a Python list a string into a Python string integers into integers and these real numbers and the floats the true values you can see get converted to a capital case true in a capital case false and null values get
Converted to a Python none value so that is how these conversions happen so now let me open back up our Python so if I print out this data variable again then run this then we can see that locate shows we have a boolean value here and
You can see that this is now uppercase and we also had a null value up here for the second objects emails as null and we can see that that got converted to none so those conversions did take place so since our top-level key here is a array
Of values that should now be equal to a list so since this is a dictionary we should be able to access this key of people and that should be a list so let’s say let’s print out the type and let’s access that people key of that
Dictionary so if I save that and print that out then you can see that that is a list so that got converted so now that we’ve converted that JSON to Python objects it’s going to be a lot easier to work with these so now we can loop
Through all these people and access each one individually so since that is a list I could say for person in data and then access that people key there and then for each one of those we could just print out the person so if I save that
And run it then you can see that it ran each of these people individually and since each of these people in our original JSON are objects themselves then those should have got converted to dictionaries as well so now we can access those values within our loop so
If we wanted to access the name of each person then we can just come here within our loop and say person and then access that name variable so if I save that and run it then you can see that now we’re accessing the names of each person in
That JSON file okay great so now that we’ve seen how to load a JSON string into a Python object now let’s do the reverse of that and dump a Python object into a JSON string and to do this we’ll use the json dot dump s method so in our example let’s
Say that we wanted to remove the phone numbers from each person and then convert this back to a json string so to remove the phone number key and value from each person within our loop we can just delete that value so within our loop here we can just say delft or
Delete person and then access that phone key and that will delete the key and the value from that dictionary so now that we’ve removed those phone numbers from those objects let’s dump all of that back into a string using that dump s method so I will say new string is equal
To and it’s JSON dump s and we want to dump in this data dictionary that we just modified so now let’s print this out so I’ll print out that new string save that and run it let me make this a little larger here so you can see that
We have a new JSON string that no longer contains the phone numbers for each person now since this is a string it would be nice if we could just format this in a way where it’s easier to read so to do this we can pass in an indent
Argument to our function to our dump s function to indent this correctly so we could say when we dump this we could pass in indent is equal to and then the number of indentions per item in the string so if we save that and run it
Then we can see that for each level it indents it twice so this nested level is ended in for x and then this one 6 x and so on so when you’re dumping these are strings if you want to format it in a way that’s very easy to read then using
That indebt method really helps out and makes it a lot easier to see exactly what’s going on now another thing that you can do to clean up your json when dumping it to a string is to sort the keys so if we pass in an argument of
Sort keys is equal to true so at the end here it’s sort under underscore keys I’ll set that equal to true if we save that and run it then we can see that now all these keys are sorted alphabetically where email comes first and then has licensed and then name now that may
Or may not be what you want but you have the option there if you want it ok so now that we’ve seen how to convert strings to Python objects and vice versa now let’s see how to load JSON files into Python objects and then write those
Objects back to JSON files so I’m going to go ahead and delete everything here that we have so far except for our imports and save that and clear that out now I have a JSON file and the same directory as my Python script here and
This is called States JSON and this is a JSON file with a list of all the US states their abbreviations and their area codes so if I wanted to load this file into a Python object then we can use the JSON load method so just remember the load method loads a file
Into a Python object and the load s method we saw before loads a string so to load in this file we first have to open it so we can open this file with a with statement so we can say with open and that file is in the same directory
So that we don’t have to pass in a path it’s just called States dot JSON and we’re just going to read that so we’ll leave that as the default and we’ll just say as F and now with this file open we can load this into a Python object by
Saying data is equal to json dot load and load in that file now if you’ve never worked with file objects before and you don’t really know what’s going on here with this with open statement and what we’re doing there then I do have a separate video on working with
File objects if you’re interested and I’ll leave a link to that in the description section below ok so now that we’ve loaded this JSON file into a Python object we should be able to loop through our data just like we did before now if I look back at the JSON file we
Can see that there is a state’s key here and then that is a list of objects and all these objects have a name abbreviation and area code so if I go back to my list to loop through those remember that was a state’s key so we
Could say for state in data then access that state’s key and then for each of those let’s just print out the entire object and that object we just called state for the state variable so if I save that and run it then it should print out that object for
Each state and now that this is a Python object it’s very easy to access this information and any way that you want so for example if you wanted to print out just the name and the abbreviation then you could say state name then we’ll put
In a comma and also print out the state abbreviation so copy that paste that in there save it and run it and you can see that now we have the state and the abbreviation access from that JSON file okay so now that we’ve loaded this JSON
From a file now let’s write this Python object out to a JSON file like we did and the smaller example let’s just remove one of the keys from the data and then write that to a new file so in this example let’s go ahead and just remove
The area codes so like we saw a bit ago to remove that from a dictionary we can just say del and then what we want to delete so we will delete the let’s see that is I believe area underscore codes yeah I’ll just copy and paste it so I
Don’t spell anything wrong so save that so now we’ve removed the area codes from that data and now to write this file as JSON we will use the JSON dump method so just like with the load and the load s method the dump method converts the data
To a JSON file and the dump s method converts the data to a JSON string just like we saw before so first we have to open the file that we want to write to so just like before when we read in the data will say with open and we can call
This whatever we want so I’ll just call this new underscore States dot JSON and we want to write this file so we gotta pass in a W as an argument there and we’ll just say as F and now we can dump that data to the file just by saying
Json dot dump and at first we want to pass in the data that we want to dump so that is this data variable here and then what we want to dump this to so in this case we want to dump this to the file we just opened which is this F variable so
If I save that and run it now let me open up my sidebar here and here in the same directory we can see that it created this new underscore States JSON so if we open this up we can see that it’s kind of compressed together but we have States key here and
Then within this list we have named Alabama a brief of briefie Asian Alabama and then it moves right on to Alaska so it did delete those phone numbers there so that did do what we wanted but it’s not really easy to read so just like we did with the dump s
Method we can also pass in an indent argument into our dump method here to clean this up a bit so if I say indent is equal to 2 and save that and run it if we open back up our new state’s JSON file here then we can see that this is
Now a lot easier to read okay so now that we’ve seen how to work with JSON strings and files and going back and forth between Python objects now let’s look at a real world example of using JSON data so that we can get a sense for
How this might be used in practice so it’s pretty common for websites to return JSON from their API s so that it’s easy to parse so let’s see what grabbing JSON data from a public API would look like and how we might parse that data so I found a Yahoo Finance API
That converts United States dollars into other currency so let’s see if we can pull down this data convert it convert the JSON into a Python object and then parse out some information so I already have a Python file here to get us started out so to make the request to the web API
I’m using the built-in URL Lib module and specifically I’m importing URL open from URL Lib dot request now you can also use the request library to do this if you’re more comfortable using that so you can see that I’ve already got the URL paste it in here to the URL open
Function and then we are setting this source variable here equal to that response and then we are using the dot read method on that response now that will just get the response from the website and right now this is just going to be a string so if I run this then it
Should just print out all the content of that website you can see it’s not very readable we’ve got newline characters in here and things like that so now let’s load this response into a Python object using the JSON module so I will create a variable down here called
Data and we’ll just say JSON dot load s since that is a string and we will pass in that source variable there I’m going to go ahead and get rid of that print statement okay and now that this is a Python object let’s immediately dump it back into a string
With the indent argument passed in so that we can clean this up a bit and see a little bit more what it looks like so I will just print out json dot dump s and we want to dump out that Python object that we just created but we’re
Gonna set indent equal to two so if we save that and run it then we can see that we’ve got some cleaned up JSON here and I’ll scroll up to the top if I can okay great so we can see we got some JSON data so we can see that the first
Key is list and then within list we have this meta key which has some metadata so we can see that it says here that the count is 188 items and also within this list we have resources and within resources there is another list in here and inside that list we just have the
Individual resources that have the conversion information so we have another resource key then we can see that it has fields and then this is where we get down to you know the name of the conversion and the price for that conversion okay so let’s start working
With this data so first of all let’s just see if that count of 188 is correct so let’s take a look we have this list and then we have these resources so let’s see if there are actually 188 resources there so I could print out I’m going to comment out our printed JSON
For now so I’m just going to print out the length of that data object and we are going to access that list and within that list we are going to access resources and just so I don’t spell this wrong I’ll copy that and paste it in there so let
Me save that and run it and you can see that we did get 188 resources in that list so that is correct okay so now let’s actually loop through all of those resources so we can say for item in data and I should have just copied and pasted
That but that’s okay that is it within list and then access resources so for the item in that list then just print out each of those items okay so that’s a little hard to see because my text is a little large here but we can see that each of these
Items has let’s see a resource key and within that resource key we have these a class name and then the fields and within those fields that’s where we have the name and the price so let’s say that we wanted to print out all the conversion names and all the prices so
To do this we can just come up here and remove this line here I’m just going to say the name is equal to our item and then to dig down in to this item we’re gonna access this resource key and then within there we’re going to access the
Fields and then within the fields we’re going to access the name so now I will save that and paste it in here because now we want to access the price so I’ll say price is equal to all that same information but now instead of the name
In those fields we want to access the price in those fields and this might look a little messy here but you’ll end up doing this a lot when working with JSON data some of this JSON data when you’re working you know with real world examples you kind of have to dig down a
Bit until you get exactly what you’re looking for so now we should have the name and the price of those conversions so if I print out name and price and save that and run it then now we have the name of each conversion and then the actual price of that conversion okay so
Now let’s actually use these conversion rates to convert US dollars to a specific currency so I’m going to create a new dictionary that has all of these names as the key and all of the conversion rates as the value so that we can access these quickly so above our
For loop here I’m just going to create an indeed empty dictionary so I’m going to say call this USD rates and set that to an empty dictionary there and then within our loop we can add each of these names as a key by saying USD rates and
We’ll use the name as the key and then the values will be the prices so we’ll set the price there and save that oh and it looks like I actually have a typo here this is supposed to be USD rates not us our rate okay so now that we have
That new dictionary set up we can access a specific value just by accessing that key so we can get the USD to Euro rate just by printing out let’s say just print out USD rate and then we want to access the key for USD into euros so
Let’s save that and run it and we can see that this says that the USD is equal to about zero point eight four six five euros now these conversions here I just have these written down so it’s not like I’ve memorized these if you pull down
This data for yourself then you can see whatever a conversion you’d like okay so now let’s actually make a conversion so if we wanted to convert let’s say 50 US dollars to euros then we could say 50 times and now this is actually a float value here but it’s coming in as a
String right now so let’s convert that to a float and then we can save that and run it so 50 US dollars would right now be about forty two point three to five euros and it changed this conversion you could just pass in a different currency
So let’s say that we wanted to convert to Indian rupees then that would just be USD to I in R so if we save that and run it then we can see that that conversion is about 50 USD to three thousand two hundred and forty now if you wanted to
Then you could write a script that pulls down this conversion information from the API once a day and then save it to a local JSON file for faster access and then write a function that accesses this information in a more repeatable way so we can see how understanding how to
Parse and work with this JSON information can be extremely beneficial for grabbing data from the internet and using it to fit our needs and there are countless other JSON API is out there and it’s essential to know how to work with this information if you want to
Grab data from external sources okay so I think that is going to do it for this video hopefully now you understand how to work with JSON data and convert back and forth between that and python objects but if anyone does have any questions about what we covered in this
Video then feel free to ask in the comment section below and I’ll do my best to answer those and if you enjoyed these tutorials and would like to support them then there are several ways you can do that the easiest ways to simply like the
Video and give it a thumbs up and also it’s a huge help to share these videos with anyone who you think would find them useful and if you have the means you can contribute through patreon and there’s a link to that page in the description section below be sure to
Subscribe for future videos and thank you all for watching you
-
Sale!
Wireless WIFI Repeater Extender Amplifier Booster 300Mbps
$29.99$14.99 Add to cartWireless WIFI Repeater Extender Amplifier Booster 300Mbps
Categories: Electronics, Wi-Fi Router, Wireless Wi-Fi Extender Tags: 300Mbps, 802.11N, Amplifier, Booster, Extender, mobile wi-fi booster, Remote, WIFI, Wireless, Wireless WIFI, Wireless WIFI Repeater, Wireless WIFI Repeater Extender, Wireless WIFI Repeater Extender Amplifier, Wireless WIFI Repeater Extender Amplifier Booster, Wireless WIFI Repeater Extender Amplifier Booster 300Mbps$29.99$14.99 -
Sale!
Full RGB Light Design Gaming Headset Headphones with Mic
$24.99$14.99 Add to cartFull RGB Light Design Gaming Headset Headphones with Mic
Categories: Electronics, Gaming, Gaming Headsets Tags: Design, Full, Full RGB Light Design Gaming Headset, Full RGB Light Design Gaming Headset Headphones, Full RGB Light Design Gaming Headset Headphones with Mic, Gamer, Gaming, Gaming Headset Headphones, gaming headset wireless, Headphone, Headphones, Headset, Light, Mic, Package, RGB$24.99$14.99 -
Sale!
Wireless BlueTooth Multi-Device Keyboard Mouse Combo
$39.99$19.99 Add to cartWireless BlueTooth Multi-Device Keyboard Mouse Combo
Categories: Electronics, Gaming, Gaming Keyboards, Keyboard Mouse Combos Tags: Combo, Keyboard, keyboard mouse combos, Mouse, MultiDevice, Set, WireKeyboard Mouse Combo, Wireless, Wireless BlueTooth Keyboard Mouse Combo, Wireless BlueTooth Keyboard Mouse Combos, Wireless BlueTooth Multi-Device Keyboard Mouse Combo, Wireless BlueTooth Multi-Device Keyboard Mouse Combos$39.99$19.99 -
Sale!
High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar
$199.99$139.99 Add to cartHigh Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar
Categories: Gaming, Gaming Chairs Tags: Adjustable, Chair, computer chairs, Desk, Executive, Gaming, Girl, Headrest, High, High Back Leather Executive Adjustable Swivel Gaming Chair, High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest, High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar, High Back Leather Executive Adjustable Swivel Gaming Chairs, Leather, Lumbar, Office, Racing, Swivel$199.99$139.99 -
Sale!
Professional LED Light Wired Gaming Headphones with Noise Cancelling Microphone
$29.99$19.99 Select optionsProfessional LED Light Wired Gaming Headphones with Noise Cancelling Microphone
SKU: N/A Categories: Electronics, Gaming, Gaming Headsets Tags: Cancelling, Gaming, Gaming Headphones with Noise Cancelling Microphone, gaming headset, Headphones, Headset, LED, Light, Mic, Microphone, Noise, Professional, Professional LED Light Wired Gaming Headphones, Professional LED Light Wired Gaming Headphones with Noise Cancelling Microphone, Wired, Wired Gaming Headphones, Wired Gaming Headphones with Noise Cancelling Microphone$29.99$19.99 -
Sale!
Gaming Desk with LED Lights USB Power Outlets and Charging Ports
$349.99$249.99 Select optionsGaming Desk with LED Lights USB Power Outlets and Charging Ports
SKU: N/A Categories: Computer Desk, Gaming, Gaming Desk Tags: and Charging Ports, Charging, Desk, Desks, Gaming, gaming desk with led lights, Gaming Desks with LED Lights, Home, LED, Lights, Monitor, Office, Outlets, Port, Power, Room, Stand, USB, USB Power Outlets, White, Workstation$349.99$249.99 -
Sale!
Wired Mixed Backlit Anti-Ghosting Gaming Keyboard
$99.99$79.99 Add to cartWired Mixed Backlit Anti-Ghosting Gaming Keyboard
Categories: Electronics, Gaming, Gaming Keyboards Tags: Antighosting, Backlit, Blue, brown, Gaming, Gaming Keyboard, gaming keyboards, gaming keyboards and mouse, Keyboard, Laptop, Switch, Wired, Wired Mixed Backlit Anti-Ghosting Gaming Keyboard, Wired Mixed Backlit Anti-Ghosting Gaming Keyboards, Wired Mixed Backlit Gaming Keyboard$99.99$79.99 -
Sale!
Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset
$119.99$59.99 Add to cartWireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset
Categories: Electronics, Gaming, Gaming Headsets Tags: 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset, ANC, Audio, Bluetooth, Cancellation, Ear, Earphone, gaming headset, Headphones, Headset, Hi-Res Over the Ear Headphones Headset, HiRes, Noise, Wireless, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Headphones, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headsets$119.99$59.99 -
Sale!
Wired Sports Gaming Headset Earbuds with Microphone
$19.99$9.99 Select optionsWired Sports Gaming Headset Earbuds with Microphone
SKU: N/A Categories: Gaming, Gaming Headsets Tags: Accessories, Earbud, Earphone, Earphones, Gaming, gaming headset with microphone, Headphones, Headset, IOS, Microphone, Sports, Wired, Wired Sports Gaming Headset Earbuds, Wired Sports Gaming Headset Earbuds with Microphone, Wired Sports Headset Earbuds$19.99$9.99 -
Sale!
150W Universal Multi USB Fast Charger 16 Port MAX Charging Station
$49.99$29.99 Add to cart150W Universal Multi USB Fast Charger 16 Port MAX Charging Station
Categories: Charging Stations, Electronics Tags: 150W, 150W Charging Station, 150W Universal Multi USB Charging Station, 150W Universal Multi USB Fast Charger 16 Port MAX Charging Station, 150W Universal Multi USB Fast Charger 16 Port MAX Charging Stations, 150W Universal Multi USB MAX Charging Station, 16 Port MAX Charging Station, 3.5A, Charger, Charging, Fast, laptop charging stations, Max, Multi, Port, Stand, Station, Universal, USB$49.99$29.99
Absolutely fantastic tutorial ❤
great video, how to do this if we have 1600 json files in one folder. Is it possible to do with one function or we need to do one by one opening them ?
How would you use a chainmap with a json response?
The original owner of Angkor Wat was an ancient Thai king which used Indian engineers and Cambodian labors to build it.
King Jayavarman VI belong to Thailand. Thailand has clear evidence that AngkorWat belongs to Thai history.
The dynasty of “Mahidharpura” centered on “Phimai” castle and belonged to the same family. (In Phimai district, Nakhon Ratchasima province, Thailand).
-King 6th Jayavarman, the founder of the “Mahitarapura” dynasty in the early 17th century B.E., located at Phimai Castle.
-King Dharnindravarman I (A brother and sister of King Jayavarman VI). Descended from the "Mahidharpura" dynasty, located at Phimai Castle.
-King Suryavarman II (Father is King Kasitindradit, mother is daughter of King Hiranyavarman [grandfather] and Queen Hiranya Lakshmi [grandmother]), who built Angkor Wat. Descended from the "Mahidharpura" dynasty, located at Phimai Castle.
I had 404 not found error how i can fix it"¿.
Nice explanation….
Thanks so much, Corey!!
cool
Amazing amazing video Corey. Just have one question. You created a new dictionary called usd_rates = dict().
After that you assigned keys and values to this dictionary using the for loop, i.e, usd_rates[name] = Price
Why didn't you use inverted commas for defining the key? Like usd_rates["name"] = Price
If you or someone else else could answer
I really like how you take the time to toss in a little tip here and there..
Most real world JSON I run into is messy and huge. A key, with a single value that’s a key that’s a single value that’s a key for a dozen times and then data.
What’s the best way to quickly explore a file to know which fields I actually want to use? Is there a tool web based or otherwise?
Helpful video, thank you, the real world example was really the cherry on the cake
Thank you Corey 😊
Great thanks for your Great work.
This is the data in json file data.json
[
{"sqlid":"100", "sqlstring":["Select * from emp_sql"]},{"sqlid":"200", "sqlstring":["Select * from sas_sql"]},{"sqlid":"200", "sqlstring":["Select * from sas_sql"]},{"sqlid":"300", "sqlstring":["Select * from sas_group"]},{"sqlid":"300", "sqlstring":["Select * from sas_group"]},{"sqlid":"300", "sqlstring":["Select * from t_sas_sql"]}
]
Please help to flatten this json data in below format
[
{"sqlid":"100","sqlstring":["Select * from emp_sql"]},
{"sqlid":"200","sqlstring":["Select * from sas_sql", "Select * from sas_sql"]},
{"sqlid":"300","sqlstring":["Select * from sas_group", "Select * from sas_group", "Select * from t_sas_sql"]}
Thanks
Thank you Corey! Legend!
Really great Corey. Working on exactly this for a python backend which I need to convert computed variables values to something that will plug into html (in other words, JS). This is so helpful!
Great tutorials. I know now how to store data as json and how to retrieve it.
excellent presentation. you posted this tutorial 5 years ago. still finding it helpful
nice content on pyton json
nice content from yt
You are simply superb. I had watched paid course videos to understand how to handle json data in python, but they are also not as clear as you explained it here. Thanks a ton!! ❤
As always another superb video
Hey all, I wrote a Library called lindex just to help with JSON objects, you can do things like index dictionaries with a list and print the JSON in a human friendly format including data key types. Check it out!
https://youtu.be/9N6a-VLBa2I?t=837
urllib.error.HTTPError: HTTP Error 404: Not Found?? 13:57
Instead of watching trash I’m watching tutorials about python. Man time to be alive
I don’t understand for what you didn’t use MySQL
can someone share the poeple_string json data here
thanks,
In the "Accessing the price" bit, is that considered data crawling?
the yahoo api is not working , giving 404 error.. I hope they have changed it maybe
Thanks for sharing!!
Great video! As someone new to Python it was easy to follow.
PERFECT explanation!
when i run the type[data] , i get dictionary instead of list and i can't iterate , any help?
6:00 indent
9:30 print different subfields
10:10 dump vs dumps dumps to file vs string
18:45 float conversion
Great video.
Thank you for taking the time to make this.
nice video
17:05 USD/TRY 3,85 😢
My heart is deeply broken!
Python Programming