Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module
- March 1, 2024
- Posted by: MainInstructor
- Category: BASIC Data Science Go Python
Video Title: Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module
Hey there how’s it going everybody in this video we’re gonna be learning how to run code in parallel using the multi processing module now if you’d also like to learn about running code concurrently using the threading module then I did recently put out a video on that as well
So I’ll be sure to leave a link to that video in the description section below now if you don’t know the difference between threading and multi processing then you should have a grasp on a difference between those once we’re finished now I would like to mention
That we do have a sponsor for this video and that is brilliant org so I really want to thank brilliant for sponsoring the video and it would be great if y’all could go and check them out using the link in the description section below and support the sponsors and I’ll talk
More about their services in just a bit so with that said let’s go ahead and get started ok so first why would we want to use multi processing so basically we want to use multi processing whenever it’s going to significantly speed up our program now the speed-up comes from different
Tasks running in parallel now in this video we’re gonna start off with a basic example of where we learn how to run some simple sleep methods in parallel but then we’ll finish up with a real-world example where we do some image processing on a directory of high
Resolution images I want to show that real world example because personally when I watch tutorials that only show how it works on basic examples then I always feel like I don’t really walk away with any useful knowledge so we’ll use the sleep method to get a good idea
Of how to use multi processing and then we’ll be sure to go over the more complicated real world example of image processing so let’s go ahead and get started so I have a starting script open here and if you’d like to follow along then I’ll be sure to have a link to this
Code in the description section below and like I said we’ll start with a very simple example to see how this works and then build up with more realistic examples so let me go over the script that I currently have open here so first I’m importing time and I’m just using
Time to measure how long it takes the script to run and that’s also what this is here this is just a start time for our script and then we have a function here called do something and all this is doing is printing that we are sleeping
For one second then we actually sleep a second using that time module and then we are printing out that we are done sleeping and then we are actually executing that function so it should do all of this and then we are calculating the finish time and printing out that our script is finished
Okay so if I run the code that we have right now we can see that it said that it was sleeping for one second done sleeping and that we finished in about one second and that sounds about right since we were running our do something function one time and that
Sleeps for one second and if we were running that function twice then our program will likely takes two seconds so let’s go ahead and see that so right below do something here I’m going to run this again and if I run that then we can
See that now that it printed out that it was sleeping for one second twice and that it took about two seconds so we can see that each time we run at this do something function it’s adding about one second to our script so our script is just waiting around sleeping for a
Second and once that’s done it moves on to run that next function and sits around waiting for another second and then at that point we’re basically done and our script finishes now I created a quick graphic to try to represent what this looks like so let me bring that up
Here in my browser real quick and this is actually the second graphic we’ll go over that in just a second okay so this is basically what it looks like for our script to be executed right now so we are running a function in this case it’s that do something function and then this
Is just coming up here and waiting and executing for one second and once that one second is over then we come back and we execute this another function and it’s that same function again so then it comes up here and executes this sleep for one second again and when that one
Second is done then we can come down here and print that our script is done and running everything in order like this is called running it synchronously now if you have some tasks that don’t need to be run synchronously then we can use the multi processing module to split
These tasks up onto other CPUs and run them at the same time now if you watch my last video on threading then I mentioned that tasks we’re going to either be I Oh bound or CPU bound so CPU bound tasks are things that are crunching a lot of numbers and using the
CPU and i/o bound tasks are things that are waiting for input and output operations to completed and they’re not really using the CPU all that much so some other examples of i/o bound tasks include file system operations and network operations like downloading stuff online now in that threading video I mentioned
That we wouldn’t get much of a speed-up when using threading on CPU bound tasks because those threads are still only running one process but with multi processing we’re going to actually spread the work out on the multiple processors on our machine and run those
Tasks on at the same time so we can use this with both IO bound tasks and CPU bound tasks so it really just depends on what we’re doing and your computer’s hardware that will determine if it’s better to use threading or multi processing but with that said let’s look
At what it looks like to run something in parallel using multi processing and I’ve got another graphic put together of what this would look like so in this example we can see that we still have our two tasks but now we’re just breaking these up onto two different processes and unlike with threading
Where we were running these concurrently and I said that running concurrently doesn’t necessarily mean that they’re running at the same time with multi processes these actually are running at the same time on different processes so we can see here that once we kick off our multiple processes and we spread out
Our tasks onto those processes then we can just run each of these functions one time and then both of these will sleep for a second and then once they’re both done we’ll come down here and print that they’re done okay so now that we’ve talked about multiply processing and
What it looks like to run code in parallel now let’s see how to actually do this with our current script so first let’s import the multi processing module so this is in the standard library so we don’t need to install anything so up here at the top
I’m just going to say import multi processing now I’m gonna show an older way of how to do multi processing first so that we can get a good idea of what’s going on but if it seems confusing at first then definitely stick around because I’m also going to show some
Newer ways of doing multi processing using pools that allow us to add this to our program with just a few lines of code okay so first instead of run do something function twice in a row like we have here let’s instead turn both of these and two processes so to do
This I’m just going to create two processes and for both of these we can just say p1 is equal to multiprocessing dot process and now we are going to pass in a target and the target is the function that we want to run so if I want to run let’s do something function
Then I can pass in do something now we want to pass in the actual function and not the return value of the function so we don’t want to execute the function with parentheses like this we just want to pass it in with no parentheses okay
So now that will be one process and now if I do a p2 is equal to multi-processing process with a target of do something then that will be our second process okay so at this point we’ve created two process objects but we’re not actually running that code so
If I run this right now then we can see that it says that it finished immediately but nothing from our function printed out so our functions act didn’t actually run so in order to get our processes to run we need to use the start method on each
One so down here below our p2 I’m going to say p1 dot start to start that first process and P 2 dot start to start that second process okay so now that will actually run our processes but it might not do exactly what we think it’ll do so
If we run this then we can see that now it runs the functions but it said that our script was finished in zero seconds and then it said that we were sleeping for one second twice because we ran that function twice and then it said that it
Was done sleeping now our entire script didn’t actually complete in zero seconds it actually took around one second but the reason that it says that it completed at zero seconds is because after it started both of these processes here while those processes were sleeping our script continued running and came
Down here and calculated out finish time and printed out that our script was finished in zero seconds and then it kicked off these processes now actually I think that I just said that it started sleeping first before it printed it out that we were finished but these processes take a little bit longer
To spin up than threads so it actually didn’t even start our processes first it actually came down here and printed that we were finished before the sleep statements even first got executed so it printed that before it said we were sleeping and then after a second we were
Done so basically what this is doing is it’s kicking off our processes but then it’s going down here and running the rest of our script before our process is finished now what if we wanted our processes to finish before we calculated the finish time and before we print it
Out that our script is finished so in order to do this we can use the join method so to do this right below start I’m gonna say p1 dot join and P 2 dot join so when we run that join method it means that the process will finish
Before moving on in the script so now if we run this then we can see that both processes started at almost the same time and then they both printed that they were done sleeping after one second and then our script continued on to print that our script had finished in
About one second now if using multi processing seems a bit complicated right now then definitely stick around until the end of the video because we’re going to see an easier way of how to do what we’re doing here but I think it’s important to understand what this is
Doing so far even if we use other methods where we don’t actually manually call these start methods and join methods ok so right now we’re not really getting that big of a speed-up now so our code ran in two seconds before and now it’s running in one second but that’s because our function
Doesn’t take too long and we’re only running it twice but what if we wanted to run our function ten times well if we were to run our code synchronously then we can take a guess that it would take ten seconds since our since one would have to finish before the other and we’d
Be running ten in a row but if we ran this with multiple then it should be significantly faster so let’s see an example of this now instead of manually creating ten different processes let’s instead create and start these in a loop so to do this
I can come up here and I’m gonna copy this whole part here and now I’m just going to overwrite all the code that we have here so far and I’m gonna say for underscore in range of ten and we’ll say P is equal to multiprocessing dot
Process with a target set to the do something function and now let’s also start that process here within our loop now if you’re unfamiliar with the underscore in Python basically that’s just a throw away variable name it’s just saying that we’re not actually using the integer from this range and
The loop so we just have something as a throwaway variable there so we’re starting all these processes here within our loop but we can’t do a P dot join within the loop because it would run join on the process before looping through and creating and starting the
Next process in the loop so it would basically be the same as running it synchronously so we need a way that we can start all of these processes in one loop and then loop through those processes again and run the join method on them so that they all finish before
The end of our script so to do this let’s just append each process to a list so above our for loop here I’m just going to create a list called processes and set that to an empty list and then below P dot start I’m going to say
Processes dot append and I will append each process to our processes list and now here below our for loop I’m going to say for process in processes let’s do a process dot join okay so just one more time here we are looping over a range of
10 so we’re going to do this loop 10 times here and each time through the loop we are creating a new process with this target of do something and we are starting that process and then we are appending that process to a processes list so then after that loop
Is complete and all of our processes have been started we’re coming through and looping over all those processes and we are joining them and again when we join a process when we run the joint method it means that it is going to wait until that finishes before continuing on
To the rest of the script so it’ll finish before it comes down here and calculates the finish time and prints that our script is finished so we’re running this do something function ten times and it sleeps for one second every time but since we’re using multiple
Processes it’ll just run all of these in parallel at the same time so instead of it taking ten seconds let’s save this and run this and see how long it actually takes so we can see that even running the function ten times we’re still finishing this in about one second
Now that might seem a little strange because I don’t actually have ten cores on this machine but your computer has ways of switching off between cores when one of them isn’t too busy so even though we had more processes than we do cores it’s still finished in about a
Second so that’s pretty good okay so now let’s look at how we can pass in arguments into our function so right now we’re running a function that doesn’t accept any arguments but let’s add a couple of arguments real quick so right now we’re just sleeping for one second
So let’s add an argument that specifies how long to sleep so up here we will accept an argument and I’m just going to pass in an argument of seconds and let’s also change that we are going to sleep for that number of seconds and let me
Also put a parenthesis there as well now this needs to be an EPS string since we’re now using this variable here within our string now I want to sleep for that number of seconds okay so with that small change our dosomething function now accepts an argument of
Seconds and then it’ll print out that we’re sleeping for that number of seconds and then it will actually sleep for that number of seconds so let’s pass in seconds as an argument to this function and we need to pass that in as a list of arguments to our process so
I’ll say down here where we are saying that our target is that do something function we can also pass in an argument of args and we’ll pass that in as a list of arguments so I’ll do 1.5 so now instead of sleeping for one second ever
For 10 different times now it’s going to sleep for 1.5 seconds for 10 different times now unlike with threads in order to pass arguments to a multi-processing process the arguments must be able to be serialized using pickle now if you don’t know what that means basically serializing something with
Pickle means that we’re converting Python objects into a format that can be deconstructed and reconstructed in another Python script so now we should expect our function to take 1.5 seconds instead so if I save this and run it then we can see that now our script is
Finishing in about 1.5 seconds ok so I said before that I was going to show you the older way of doing multi processing and then I’d show you what I believe is a faster easier way of doing this and I still wanted to show you the manual way
Of creating these processes because I think this can still be useful depending on what you’re doing and also I think it’s better to learn this manual way first to understand a bit better what’s going on in the background but in Python 3.2 they added something called a
Process pull executor and in a lot of cases this will be an easier and more efficient way to run multiple processes and it also allows us to easily switch over to using multiple threads instead of processes as well depending on the problem that we’re trying to solve so
Let’s replace what we’re current what we currently have and instead use at this process pull executor now this actually isn’t in the multi processing module it’s in the concurrent futures module instead so up here at the top let’s instead import concurrent dot futures and I actually don’t think I need multi
Processing anymore so I’m just going to say import concur dot futures now I’m gonna leave everything else that I have here for now so that we can see the difference between these now when we use this process pull executors it’s usually best to use this with a context manager so
Above our processes list here I’m going to do the same thing that we already have but just with our concurrent futures module instead so I’m going to say with concurrent dot futures dot process pool executors and make sure you get those capitalizations in the right place and then we will say whoops will
Say as executor and now within our or with our executor here there are a couple of different methods that we can use now if we want to execute the function once at a time then we can use the submit method so the submit method schedules a function to be executed and
Returns a future object so let’s add this in and then I’ll explain this a bit more so I’m going to say F 1 is equal to executors dot submit and I will submit that do something function and let’s also pass in an argument of one so again the submit method schedules a function
To be executed and returns a future object so a future object basically encapsulates the execution of our function and allows us to check on it after it’s been scheduled so we can check that it’s running or if it’s done and also check the result so if we grab
The result then it’ll give us the return value of the function now right now we’re just printing out values but let me add in a return value so that we can grab that so instead of just printing that we are done sleeping up here instead I’m going to return that string
So I’m going to say return done sleeping instead of just printing that out okay so now that’s returning that string so if we still want to print that out then we’ll need to print that return value so let’s grab that by using the result method on that future object so
I’m gonna say print and we will print out f1 dot result now if we run the return method then this will wait until the function completes okay so let’s comment out while we had before and run our code so I’m gonna comment out this processes list here and
Our previous starts and joins and instead we’re just going to use this process pull executors okay so if I run this then we can see that that still works and that’s a lot less code than we had down here that’s commented out but we’re still not running this multiple
Times yet like we were down here so if we wanted to run this multiple times then we could just run submit multiple times so I could say let me go above our result here I’m gonna add in another execution of this do something function
So I’m going to call this f2 is equal to executors not submit do something with one second and then I will also print out the f2 result so if I run this then we can see that it’s the same thing it kicks both of these off at the same time
And we finished in about one second and if we wanted to run this ten times like we did below then we likely wouldn’t want to run submit ten two different times so we could use a loop like we did before so instead of running one at a
Time I’m gonna use a loop and we could use a regular loop like we did below but I’m gonna go ahead and use a list comprehension to create these instead so we could say I’m just going to copy this executors dot submit part and I’m just going to overwrite all of this other
Stuff right now and I’m gonna say results are equal to then I will start a list comprehension here and say executors dot submit do something for one second for underscore range ten now if you’re not familiar with list comprehensions like we have here then i
Do have a separate video on that as well so I’ll put a link to that in the description section below if you’ve never seen this type of code before and if you’re not comfortable using list comprehensions then you can always use a regular loop like
We did down here below okay so now we’ve created a list comprehension that’s running our submit method with this do something function and an argument of one second ten different times now in order to get these results we can actually use another function from the concurrent futures module called as
Completed and this will give us an iterator that we can loop over that will yield the results of our processes as they’re completed so I think this is really useful and it’s one of the good things about using these processing pool executors so to use this we can say just
For F in concurrent oops sorry about my typing their concurrent dot futures dot as underscore completed and now we want to pass in this list of results which is a list of futures objects and now within this list let’s print out F dot result
So if we run this oops and it looks like I have some invalid syntax here oh I forgot to say I should have said for underscore in range of ten some of you probably saw that as I was typing it out okay so now if I run this then we can
See that we slept for one second now it still ran ten different times but if we scroll down to the bottom then we can see how much time it took so we can see here that it actually took three seconds this time now the reason behind that is
That our pool may have made a decision based on our hardware not to a lot as many processes so that’s why it might take longer but even though it took longer in our simple little example here I still you like to use these processing pool executors most of the time because
I trust it too a lot the processes a lot more than I trust myself so I pass that off to the process pool executors to do and to make that decision for me now to prove that these results are actually coming in as they’re completed let me
Actually pass in a different range of seconds for our processes to sleep and those should print out and the order that they complete so I’m gonna create a list of seconds to sleep and I’ll make that sleep from five seconds all the way down to one second
So above our results here I’m going to make a list of seconds and I will just make a list of five four three two one and I’ll also print out the seconds in the return statement of our do something function so that we can tell which ones are finishing and in what order
So again I’m going to make this an F string by putting an F right there and I’m just going to pass that in two so we can see which seconds is actually done sleeping so now let me also change our list comprehension here so that we are running or do something function with
Each of these seconds in this seconds list so I’m gonna say executors dot submit do something and I want to do that for whatever second so for SEC in our seconds list okay so now if I run this then we can see that it actually started our five-second process first
And then our four then our 3 then our two then our one but it finished these in the order that they came in and the lowered number seconds are towards the top now I’m not sure why the 1 second process took so much longer than the 2
And the 3 second processes I guess it just got hung up on something but the 4 and the 5 second processes were down here at the bottom actually let me run that one more time and oh okay so that’s why it’s because our 1 second process
Was down here and since I have 4 cores on my machine it started these four processes here first and it didn’t start the 1 second process until this 2 was finished right here so that’s why that took a little bit longer but since we are using this as completed method this
Actually did print our result out our results in the order that they completed so this two second one finished first and then this 3 then 1 then 4 then 5 and can see here down at the bottom that our script is still finishing in about five seconds so that’s pretty good okay so
With this submit method right now it’s submitting each function once at a time but in order to run submit on an entire list then we need to do a loop or a comprehension like we did here but if you’re familiar with the built-in map method then there is actually something
Similar that we can do with processes where we can use the map method to run our function over a list of values so if you’re familiar with the built-in map method then this is very similar except it uses processes instead so it runs the function every time or with every item
Of the integral that we pass in so let’s say that I want to map our function to our list of seconds so to do this I am just going to overwrite our list comprehension here and I’m not going to be using this as completed anymore
Either so I’m also going to get rid of that so now in order to do this we can simply say executors dot map and now we will map our dosomething function and we will map our list of seconds so again what this map method does if you’re not
Familiar with the built-in Python map method and what it does basically map will run this do something function with every item of this list with every item of whatever iterable you pass in so that is what map does now when we were using the submit method it returned future
Objects but when we use map it just returns the results now it is going to run those processes in parallel but instead of returning the results as they’re completed like we saw before map is going to return the results and in the order that they were started so to
Loop over these results we can simply just do a for loop so I’m going to say for result in results and then I will just print out our result look make sure that I’m printing out the result and not that result list okay so now if I run this then we can
See that all of our processes kicked off at pretty much the same time except for that one second one it looked like it got outside of the pool like it did before but they actually didn’t all complete at the same time but when you loop over your results using map like we
Did here then it returns the results and the order that they were started so since we slept for five seconds first then we waited for that one to finish before printing out the other results but it still didn’t slow us down we can see that our entire script still
Finished in five seconds here but it looks like our five seconds was done sleeping first and then our four then three then two then one it actually didn’t finish in that order but it printed out in that order because again it prints out the ones that in the order
That they were started and not in the order that they were finished now another thing to point out here is that if our function raised it as an exception it won’t actually raise that exception while running the process the exception will be raised when its value is retrieved from the results iterator
So if you need to handle any exceptions then you can do that here within the iterator if you’d like and if you’d like to learn more about handling exceptions then I do have a more in-depth video if you’d like to learn more about that so
I’ll be sure to leave a link to that video in the description section below for anyone who’s interested now even if we don’t grab our results within the context manager it’s still going to automatically join all of those processes and let them finish after the context manager ends so if I comment out
Where we are printing out our results let me also get rid of our old way of doing this here so that we can see our code below our processes here so now I’m commenting out where we’re printing those results and if I run this then we
Can see that it still didn’t move on to our finish time here until it returned the results from our processes so any time you’re using a context manager like this it’s going to automatically join those processes and they’re going to complete for that context manager finishes okay
So now that we’ve looked at a basic example of using sleep now let’s take a look at a more real-world example of we’re using multiple processes would be useful so I’ve got another script open here where I’m not using multiple processes at the moment so let me open
This up and let’s go over what this is doing and again I’ll have a link to this code in the description section below for anyone who wants to follow along so in our last video on threading I showed how we could use threads to go out and download some high resolution photos
From unsplash now if you don’t know what unsplash is it’s a website that has some really nice photos available for anyone to use now downloading images is something that is io bound since we’re waiting for the images to download so for this multi-processing example we want something that is more CPU bound so
In this script I’m doing some image processing on the images that we downloaded in the threading video and I’ll have these images available in the description section below as well in case anyone didn’t follow along with that threading video so let me go over this script to show you how someone
Might do this image processing normally so I’m using the pillow library here which if you don’t know what the pillow library is it’s a image library for Python that makes it makes image processing easy I also have a video on pillow if you’d like to learn more about
Image processing but I have a list here of image names and these are all of the images or all of the image names that we downloaded in the last video using threading so I am starting a counter here so that we can measure how long our
Script took I am setting a size of 1200 pixels and that is what we are going to resize these high rollaz resolution photos to okay and now what I am doing is I am looping over all of those image names so with one image at a time I am
Opening that image and then I am running running this image filter and we’re just doing a Gaussian blur on that image and then I am doing an image thumbnail and setting it to this 1200 pixel size then we are saving that image into a processed folder with that the image
Name and then we are printing out that that image was processed now I do have a processed folder here within my directory if you don’t you’ll probably get an error and I also have all of these photos here within my directory if you don’t have those then you’re also
Going to get an error so you want to have all of those photos in order for this code to work then after that’s done I am printing out the time that it took and then I am printing out that our script is finished okay so this is processing 15 high-resolution photos so
If I run this now then let’s see how long this takes so I’m running this we can see that it is just going through synchronously and running these one at a time and then we will print out or the final time once this is done now I think
That my processing here I think that this still might be more i/o bound than CPU bound because opening the image and saving this here is going to be more of an i/o bound thing I don’t know how much is actually getting a CPU bound here using this filter but if we were doing
Some more computations then this would be more CPU bound and we can even test this in a second but anyways we can see that our script finished in 22 seconds so that’s how long it took to do this image processing here on 15 of these high resolution photos now when we’re
Processing a lot of items like this this is actually a great candidate to use multi processing because this can be one of those CPU bound operations that’s spending a lot of time processing each image and it’s not moving on to the next image until the previous one finishes
And if we use multiple processes then it can actually go ahead and move on to the to process another image while the previous one is finishing and even if this is i/o bound unlike threading CPR multi-processing is actually going to be beneficial for i/o bound processes as well so now let’s see
How we can change this code so that it’s using multiple processes instead so first let’s think about what we’re doing here so we’re looping over our list of images and processing each one at a time so if we remember from our previous examples then this would probably be a good
Candidate for the processing pull map method where we can pass in a function and a list and have that function run with every value in that list but first we’ll have to create a function that will process a single image so to do that if we think about this we can
Basically just replace our for loop because our for loop is processing a single image from our images list so instead we can just change this one line here and just turn this into a process image function and we will just accept this image name as the argument and now
That we have that function that processes a single image then we can create a process pool and map our list of images using that function so first let’s import the concurrent futures module so that we can use that so up here at the top I’m going to import
Concurrent dot futures and now down here below our function we can do this by saying with concurrent dot futures dot process pull executors as executors and now within our context manager we can just say executors dot map and we want to map this process image function and
We want to pass in the list of these image names now just in case that was confusing let me go over exactly what we did again in order to change our previous script in order to use multi-processing so we were using a for loop here but instead we changed this to a function
That processes one image at a time and now we’re using our process simple executors like we saw before within a context manager we’re running executor dot map and we are passing in that function that we created and then we’re passing in our list of image names and again what map
Does is it runs this process image function with every item in this image names list and just with those small changes this will actually use multiple processes to process those images and run these in parallel instead of running these synchronously so if I run this whoops and I made another mistake here
You guys probably saw that one also but I needed to put a colon there okay hopefully that’s my only mistake okay so if I run this then we can see now these are processing faster and I’m gonna open up my activity monitor here and scroll
Down to the peas and we can see here that we have multiple Python processes running here so we can actually see these in our activity monitor and now that this script is finished then we can see most of those go away the only one that’s still here is probably just one
That is open here in my sublime text so we could see all of those kick off in our Activity Monitor and we could see that they were all returning faster that’s because they were running in parallel and now instead of taking 22 seconds like it did before now it
Finished in seven seconds so more than a third of the time so that’s a pretty significant speed up there and this would be even more significant if we were processing even more images so the speed ups can be pretty drastic depending on what you’re doing now again
You might want to experiment when using multiple threads or processes for specific tasks depending on what you’re doing and what kind of hardware you’re running one might be drastically different than the other and it’s hard to tell exactly what you should be using without some benchmarks but again a good
Rule of thumb is that you want to use threads for things that are i/o bound and you’ll want to use processes for things that are CPU bound but you may actually see significant speed ups using threads with this example since this is reading and writing to disk which is an
I/o bound operation now one nice thing about using the concurrent futures library like this is that switching between threads and processes is just as simple as changing this process pull executors and just using a thread pull executors instead and you can do that vice versa to change from threads to processes with your
Programs so if I change this to a thread pool executors let me see if I’m right about a lot of this being IO bound here if I run this then we’ll see how long this takes us to finish this script using threads instead of multiple processes and that was actually faster
So that was 7.2 seconds so even though I tried to put together an example that was doing a lot of processing on the CPU it looks like most of this is IO bound from opening and saving these files and not CPU bound but from doing some image processing here with these Ghazni and
Blurs and these resizes and things like that but that’s ok that’s why you want to always experiment and you know if you try it with a process pull then maybe try it once with threads and see if you get better execution and also whenever you add in even more items maybe
Processes will start to become more performance than threads so it really just depends on what you’re doing now before we finish up here I’d like to mention the sponsor of this video and that is brilliant org so when we’re talking about threading and multi-processing these topics are
Especially useful in the field of data science and the data science field is growing at a very rapid pace if you’d like to learn more about programming and data science then I would definitely recommend checking out brilliant org so brilliant is a problem-solving website that helps you understand underlying
Concepts by actively working through guided lessons and they’ve recently added some brand-new interactive content that makes solving puzzles and challenges even more fun and hands-on and if you’d like to learn more about data science and programming with Python then I would recommend checking out their new probability course that covers
Everything from the basics to real-world applications and also fun things like casino games they even use Python in their statistics courses and will quiz you on how to correctly analyze the data within the language so they’re guided lessons will challenge you but you also have the ability to get hints
Or even solutions if you need them it’s really tailored towards understanding the material they even have a coding barment built into their website so that you can run code directly in the browser and that is a great compliment to watching my tutorials because you can apply what you’ve learned in their
Active problem-solving environment and that helps to solidify that knowledge so to support my channel and learn more about brilliant you can go to brilliant org forge / CMS to sign up for free and also the first 200 people that go to that link will get 20% off the annual
Premium subscription and you can find that link in the description section below and again that is brilliant dot org ford slash CMS ok so I think that’s going to do it for this video I hope you feel like you’ve got a good idea of how to use the multi-processing module and
How we can use these to speed up our scripts now I hope you also feel like you got a good overview of threads and processes and when you should use those but like I said if you’re unsure then there’s no hurt and simply trying both
On a subset of your data to see what gives you the most speed up now there are some more advanced topics that we could cover in future videos such as race conditions locks and things like that but we’ll save that for a future video if anyone is interested but if
Anyone has any questions about what we covered in this video then feel free to ask in the comment section below and I’ll do my best to answer those and if you enjoy these tutorials and would like to support them then there are several ways you can do that the easiest ways to
Simply like the video and give it a thumbs up and also it’s a huge help to share these videos with anyone who you think would find them useful and if you have the means you can contribute through patreon and there’s link to that page in the description section below be
Sure to subscribe for future videos and thank you all for watching you you
-
Sale!
Wireless WIFI Repeater Extender Amplifier Booster 300Mbps
$29.99$14.99 Add to cartWireless WIFI Repeater Extender Amplifier Booster 300Mbps
Categories: Electronics, Wi-Fi Router, Wireless Wi-Fi Extender Tags: 300Mbps, 802.11N, Amplifier, Booster, Extender, mobile wi-fi booster, Remote, WIFI, Wireless, Wireless WIFI, Wireless WIFI Repeater, Wireless WIFI Repeater Extender, Wireless WIFI Repeater Extender Amplifier, Wireless WIFI Repeater Extender Amplifier Booster, Wireless WIFI Repeater Extender Amplifier Booster 300Mbps$29.99$14.99 -
Sale!
Full RGB Light Design Gaming Headset Headphones with Mic
$24.99$14.99 Add to cartFull RGB Light Design Gaming Headset Headphones with Mic
Categories: Electronics, Gaming, Gaming Headsets Tags: Design, Full, Full RGB Light Design Gaming Headset, Full RGB Light Design Gaming Headset Headphones, Full RGB Light Design Gaming Headset Headphones with Mic, Gamer, Gaming, Gaming Headset Headphones, gaming headset wireless, Headphone, Headphones, Headset, Light, Mic, Package, RGB$24.99$14.99 -
Sale!
Wireless BlueTooth Multi-Device Keyboard Mouse Combo
$39.99$19.99 Add to cartWireless BlueTooth Multi-Device Keyboard Mouse Combo
Categories: Electronics, Gaming, Gaming Keyboards, Keyboard Mouse Combos Tags: Combo, Keyboard, keyboard mouse combos, Mouse, MultiDevice, Set, WireKeyboard Mouse Combo, Wireless, Wireless BlueTooth Keyboard Mouse Combo, Wireless BlueTooth Keyboard Mouse Combos, Wireless BlueTooth Multi-Device Keyboard Mouse Combo, Wireless BlueTooth Multi-Device Keyboard Mouse Combos$39.99$19.99 -
Sale!
High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar
$199.99$139.99 Add to cartHigh Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar
Categories: Gaming, Gaming Chairs Tags: Adjustable, Chair, computer chairs, Desk, Executive, Gaming, Girl, Headrest, High, High Back Leather Executive Adjustable Swivel Gaming Chair, High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest, High Back Leather Executive Adjustable Swivel Gaming Chair with Headrest and Lumbar, High Back Leather Executive Adjustable Swivel Gaming Chairs, Leather, Lumbar, Office, Racing, Swivel$199.99$139.99 -
Sale!
Professional LED Light Wired Gaming Headphones with Noise Cancelling Microphone
$29.99$19.99 Select optionsProfessional LED Light Wired Gaming Headphones with Noise Cancelling Microphone
SKU: N/A Categories: Electronics, Gaming, Gaming Headsets Tags: Cancelling, Gaming, Gaming Headphones with Noise Cancelling Microphone, gaming headset, Headphones, Headset, LED, Light, Mic, Microphone, Noise, Professional, Professional LED Light Wired Gaming Headphones, Professional LED Light Wired Gaming Headphones with Noise Cancelling Microphone, Wired, Wired Gaming Headphones, Wired Gaming Headphones with Noise Cancelling Microphone$29.99$19.99 -
Sale!
Gaming Desk with LED Lights USB Power Outlets and Charging Ports
$349.99$249.99 Select optionsGaming Desk with LED Lights USB Power Outlets and Charging Ports
SKU: N/A Categories: Computer Desk, Gaming, Gaming Desk Tags: and Charging Ports, Charging, Desk, Desks, Gaming, gaming desk with led lights, Gaming Desks with LED Lights, Home, LED, Lights, Monitor, Office, Outlets, Port, Power, Room, Stand, USB, USB Power Outlets, White, Workstation$349.99$249.99 -
Sale!
Wired Mixed Backlit Anti-Ghosting Gaming Keyboard
$99.99$79.99 Add to cartWired Mixed Backlit Anti-Ghosting Gaming Keyboard
Categories: Electronics, Gaming, Gaming Keyboards Tags: Antighosting, Backlit, Blue, brown, Gaming, Gaming Keyboard, gaming keyboards, gaming keyboards and mouse, Keyboard, Laptop, Switch, Wired, Wired Mixed Backlit Anti-Ghosting Gaming Keyboard, Wired Mixed Backlit Anti-Ghosting Gaming Keyboards, Wired Mixed Backlit Gaming Keyboard$99.99$79.99 -
Sale!
Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset
$119.99$59.99 Add to cartWireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset
Categories: Electronics, Gaming, Gaming Headsets Tags: 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset, ANC, Audio, Bluetooth, Cancellation, Ear, Earphone, gaming headset, Headphones, Headset, Hi-Res Over the Ear Headphones Headset, HiRes, Noise, Wireless, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Headphones, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headset, Wireless Bluetooth 5.3 ANC Noise Cancellation Hi-Res Over the Ear Headphones Headsets$119.99$59.99 -
Sale!
Wired Sports Gaming Headset Earbuds with Microphone
$19.99$9.99 Select optionsWired Sports Gaming Headset Earbuds with Microphone
SKU: N/A Categories: Gaming, Gaming Headsets Tags: Accessories, Earbud, Earphone, Earphones, Gaming, gaming headset with microphone, Headphones, Headset, IOS, Microphone, Sports, Wired, Wired Sports Gaming Headset Earbuds, Wired Sports Gaming Headset Earbuds with Microphone, Wired Sports Headset Earbuds$19.99$9.99 -
Sale!
150W Universal Multi USB Fast Charger 16 Port MAX Charging Station
$49.99$29.99 Add to cart150W Universal Multi USB Fast Charger 16 Port MAX Charging Station
Categories: Charging Stations, Electronics Tags: 150W, 150W Charging Station, 150W Universal Multi USB Charging Station, 150W Universal Multi USB Fast Charger 16 Port MAX Charging Station, 150W Universal Multi USB Fast Charger 16 Port MAX Charging Stations, 150W Universal Multi USB MAX Charging Station, 16 Port MAX Charging Station, 3.5A, Charger, Charging, Fast, laptop charging stations, Max, Multi, Port, Stand, Station, Universal, USB$49.99$29.99
Hey everybody! I hope you find this video helpful. I'm finally moved into my new home, so I should be back on a schedule of regular releases very soon. Hope everyone has a great weekend!
Great video. Thanks.
Thank you Corey, it is brilliant to start from the basics. Really helpful approach and advice !
Fantastic explanation!
Hello, thank you for the amazing tutorial . I have one question please. Most multiprocessing tutorials use the if statement if _name_ == '__main__' before doing multiprocessing. Can you guide us if that is necessary?
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if _name_ == '__main__':
freeze_support()
…
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Can anyone please help me? I have this RuntimeError
Why is separated loop for the 'joins' necessary?
my new favorite python teacher! TY
not working
Superclear.
The faster way would be to launch multiple processes and then launch multiple threads inside those processes.
Thanks man. Very nice!👌
hi there
hello
I learned Flask and SQLAlchemy from this guy. I know this video will be good.
Thank you for the video 🙂
Huge benefits. Thank you.
i love the sound you made at 5:51
This video is gold, every second of it. You are very talented at teaching complex concepts, you can't even begin to fathom how much I appreciate your work. Thanks!
I have 38 files that I pass through a function. Each file takes 1-2 minutes to finish. I'm using a core i5 laptop with 4 CPU cores. I've re-written to take advantage of multiprocessing. Now, all 38 files are finished in less than 20 minutes. That's the good news. The bad is that it maxes out my pc to 100% of the CPU when running.
I had expected it to start 1 file per CPU core and finish that file before starting the next. (Its looping through the directory). But in reality, it seems to start all 38 files. It seems to process a little of each until finished. The time stamp on the files indicates almost all were created within the same minute.
My question is this: can I force Python to finish a file before starting the next? (each file is processed on a different CPU core…)
If this possible, what is it called? I mean, how do I research it?
Thank you
JD
time.sleep(seconds) – is a bad example to explain multiprocessing, as long as it behaves just like IO-bound. So we would not see any difference in performance between threading and multiprocessing modules. Instead of using sleep – use some random math calculations in a loop of, dunno, 5_000_000 iterations. In this case we are running 100% CPU-bound operations and in this case, threading will perform way more slower than multiprocessing. That's the only inaccuracy which i found by watching the video. Thanks for the video!
ok but how can we run two different code files at the same time in parallel ?
I am getting RuntimeError while running multiprocessing eg
So helpful. Thank you !
add these codes to run for window user:
if _name_ == '__main__':
Thank you, Corey, for this awesome tutorial. It was my intro to multiprocessing and I have already put it to practice to my benefit. I have one request if possible: please do an advanced version of this and explain chunksize, imap, imap_unordered, etc. I'm confused, e.g., it seems like with chunksize =1, map and imap_unordered behave in the same way. The documentation of this module is not great.