Multiprocessing in Python | Set 1 (Introduction)

Multiprocessing allows a Python program to run multiple independent processes concurrently. Each process has its own memory space and can run on a separate CPU core, making it useful for CPU-intensive tasks. It is commonly used when:

• A task requires significant CPU computation.
• Multiple independent tasks need to run at the same time.
• The program needs to use multiple CPU cores.
• A large task can be divided into smaller independent tasks.

Process Management in Python

Creating Processes

Python's multiprocessing module provides the Process class for creating independent processes. A target function can be assigned to each process, along with any required arguments.

import multiprocessing
def print_cube(num):
    print("Cube: {}".format(num * num * num))

def print_square(num):
    print("Square: {}".format(num * num))

if __name__ == "__main__":
    p1 = multiprocessing.Process(target=print_square, args=(10, ))
    p2 = multiprocessing.Process(target=print_cube, args=(10, ))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print("Done!")

Square: 100
Cube: 1000
Done!

Explanation:

• Process() creates a new process and assigns a target function.
• target specifies the function to execute, while args passes arguments to it.
• start() begins process execution.
• join() waits for the process to finish before continuing.
• p1 calculates the square and p2 calculates the cube.
• Done! is printed after both processes complete.

Checking Process ID and Status

Each process has its own process ID and execution status. Python provides methods and attributes to identify a process and check whether it is still running.

import multiprocessing
import os

def worker1():
    print("ID of process running worker1: {}".format(os.getpid()))

def worker2():
    print("ID of process running worker2: {}".format(os.getpid()))

if __name__ == "__main__":
    print("ID of main process: {}".format(os.getpid()))

    p1 = multiprocessing.Process(target=worker1)
    p2 = multiprocessing.Process(target=worker2)

    p1.start()
    p2.start()

    print("ID of process p1: {}".format(p1.pid))
    print("ID of process p2: {}".format(p2.pid))

    p1.join()
    p2.join()

    print("Both processes finished execution!")

    print("Process p1 is alive: {}".format(p1.is_alive()))
    print("Process p2 is alive: {}".format(p2.is_alive()))

ID of main process: 18
ID of process running worker1: 19
ID of process running worker2: 20
ID of process p1: 19
ID of process p2: 20
Both processes finished execution!
Process p1 is alive: False
Proce...

Explanation:

• os.getpid() returns the ID of the process currently executing the function.
• p1.pid and p2.pid give the IDs assigned to the created processes.
• The process IDs of worker1() and worker2() match their corresponding process objects.
• join() waits for both processes to finish.
• is_alive() checks whether a process is still running.
• After join(), both processes have finished, so is_alive() returns False.

Memory Isolation Between Processes

Each process has its own memory space. A normal global variable changed inside a child process does not change the corresponding variable in the main process.

import multiprocessing
result = []

def square_list(mylist):
    global result
    for num in mylist:
        result.append(num * num)

    print("Result(in process p1): {}".format(result))
if __name__ == "__main__":
    mylist = [1, 2, 3, 4]

    p1 = multiprocessing.Process(target=square_list, args=(mylist,))
    p1.start()
    p1.join()

    print("Result(in main program): {}".format(result))

Result(in process p1): [1, 4, 9, 16]
Result(in main program): []

Explanation:

• result is initially an empty list in the main process.
• The child process modifies its own copy of result.
• The main process still has its original empty list.
• Data must be explicitly shared when processes need to access the same data.

Sharing Data Between Processes

Multiprocessing module provides shared-memory objects and managers for sharing data between processes.

Sharing Data Using Array and Value

Array and Value provide shared memory that can be accessed by multiple processes.

import multiprocessing
def square_list(mylist, result, square_sum):
    for idx, num in enumerate(mylist):
        result[idx] = num * num

    square_sum.value = sum(result)

    print("Result(in process p1): {}".format(result[:]))
    print("Sum of squares(in process p1): {}".format(square_sum.value))

if __name__ == "__main__":
    mylist = [1, 2, 3, 4]

    result = multiprocessing.Array('i', 4)
    square_sum = multiprocessing.Value('i')

    p1 = multiprocessing.Process(
        target=square_list,
        args=(mylist, result, square_sum)
    )

    p1.start()
    p1.join()

    print("Result(in main program): {}".format(result[:]))
    print("Sum of squares(in main program): {}".format(square_sum.value))

Result(in process p1): [1, 4, 9, 16]
Sum of squares(in process p1): 30
Result(in main program): [1, 4, 9, 16]
Sum of squares(in main program): 30

Explanation:

• Array('i', 4) creates shared memory for four integers.
• Value('i') creates shared memory for one integer.
• The child process stores the calculated squares in result.
• square_sum.value stores the sum of the squares.
• The main process can access the updated values after the child process finishes.

Sharing Python Objects Using Manager

multiprocessing.Manager() can create shared objects such as lists and dictionaries that can be accessed by multiple processes.

import multiprocessing
def print_records(records):
    for record in records:
        print("Name: {0}\nScore: {1}\n".format(record[0], record[1]))
def insert_record(record, records):
    records.append(record)
    print("New record added!\n")
if __name__ == "__main__":
    with multiprocessing.Manager() as manager:
        records = manager.list([
            ('Sam', 10),
            ('Adam', 9),
            ('Kevin', 9)
        ])
        new_record = ('Jeff', 8)
        p1 = multiprocessing.Process(
            target=insert_record,
            args=(new_record, records)
        )
        p2 = multiprocessing.Process(
            target=print_records,
            args=(records,)
        )
        p1.start()
        p1.join()
        p2.start()
        p2.join()

New record added!

Name: Sam
Score: 10

Name: Adam
Score: 9

Name: Kevin
Score: 9

Name: Jeff
Score: 8

Explanation:

• Manager() creates a manager for shared objects.
• manager.list() creates a list that can be accessed by multiple processes.
• p1 adds a new record to the shared list.
• p2 reads and prints the updated list.
• Manager objects support Python data types such as lists and dictionaries but have more overhead than shared-memory objects.

Communication Between Processes

Processes may need to exchange data or messages while performing separate tasks. Python provides Queue and Pipe for communication between processes.

Using Queue for Communication

A Queue allows processes to safely exchange data. The put() method adds data, while get() retrieves it.

import multiprocessing
def square_list(mylist, q):
    for num in mylist:
        q.put(num * num)
def print_queue(q):
    print("Queue elements:")
    while not q.empty():
        print(q.get())
    print("Queue is now empty!")
if __name__ == "__main__":
    mylist = [1, 2, 3, 4]
    q = multiprocessing.Queue()
    p1 = multiprocessing.Process(
        target=square_list,
        args=(mylist, q)
    )
    p2 = multiprocessing.Process(
        target=print_queue,
        args=(q,)
    )
    p1.start()
    p1.join()
    p2.start()
    p2.join()

Queue elements:
1
4
9
16
Queue is now empty!

Explanation:

• Queue() creates a communication queue.
• put() adds calculated values to the queue.
• get() retrieves values from the queue.
• The first process adds the squares, and the second process reads them.

Using Pipe for Communication

A Pipe provides two connection objects for communication between two processes. The send() method sends data, while recv() receives it.

import multiprocessing
def sender(conn, msgs):
    for msg in msgs:
        conn.send(msg)
        print("Sent the message: {}".format(msg))

    conn.close()
def receiver(conn):
    while True:
        msg = conn.recv()

        if msg == "END":
            break

        print("Received the message: {}".format(msg))
if __name__ == "__main__":
    msgs = ["hello", "hey", "hru?", "END"]

    parent_conn, child_conn = multiprocessing.Pipe()

    p1 = multiprocessing.Process(
        target=sender,
        args=(parent_conn, msgs)
    )

    p2 = multiprocessing.Process(
        target=receiver,
        args=(child_conn,)
    )

    p1.start()
    p2.start()

    p1.join()
    p2.join()

Sent the message: hello
Sent the message: hey
Sent the message: hru?
Sent the message: END
Received the message: hello
Received the message: hey
Received the message: hru?

Explanation:

• Pipe() creates two connected communication endpoints.
• send() sends a message through one end.
• recv() receives a message from the other end.
• The receiver continues until it receives the "END" message.
• A pipe is suitable when communication is required between two processes.