changeo.Multiprocessing

Multiprocessing functions

class changeo.Multiprocessing.DbData(key, records)

Bases: object

A class defining IgRecord data objects for worker processes

class changeo.Multiprocessing.DbResult(key, records)

Bases: object

A class defining IgRecord result objects for collector processes

data_count
changeo.Multiprocessing.collectDbQueue(alive, result_queue, collect_queue, db_file, task_label, out_args, add_fields=None)

Pulls from results queue, assembles results and manages log and file IO

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
  • result_queue – multiprocessing.Queue holding worker results
  • collect_queue – multiprocessing.Queue to store collector return values
  • db_file – Database file name
  • task_label – Task label used to tag the output files
  • out_args – Common output argument dictionary from parseCommonArgs
  • add_fields – List of fields added to the writer not present in the in_file; if None do not add fields
Returns:

Adds a dictionary with key value pairs to collect_queue containing

‘log’ defining a log object, ‘out_files’ defining the output file names

Return type:

None

changeo.Multiprocessing.feedDbQueue(alive, data_queue, db_file, group_func=None, group_args={})

Feeds the data queue with Ig records

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues if False exit process
  • data_queue – multiprocessing.Queue to hold data for processing
  • db_file – Database file
  • group_func – Function to use for grouping records
  • group_args – Dictionary of arguments to pass to group_func
Returns:

None

changeo.Multiprocessing.processDbQueue(alive, data_queue, result_queue, process_func, process_args={})

Pulls from data queue, performs calculations, and feeds results queue

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
  • data_queue – multiprocessing.Queue holding data to process
  • result_queue – multiprocessing.Queue to hold processed results
  • process_func – Function to use for filtering sequences
  • process_args – Dictionary of arguments to pass to process_func
Returns:

None