changeo.Multiprocessing

Multiprocessing functions

class changeo.Multiprocessing.DbData(key, records)

Bases: object

A class defining data objects for worker processes

id

result identifier

data

list of data records

valid

True if preprocessing was successful and data should be processed

class changeo.Multiprocessing.DbResult(key, records)

Bases: object

A class defining result objects for collector processes

id

result identifier

data

list of original data records

results

list of successfully processed records

failed

list of records that failed processing for workers than may split sets

valid

True if processing was successfull and results should be written

log

OrderedDict of log items

data_count
changeo.Multiprocessing.collectDbQueue(alive, result_queue, collect_queue, db_file, task_label, out_args, add_fields=None)

Pulls from results queue, assembles results and manages log and file IO

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
  • result_queue – multiprocessing.Queue holding worker results
  • collect_queue – multiprocessing.Queue to store collector return values
  • db_file – Database file name
  • task_label – Task label used to tag the output files
  • out_args – Common output argument dictionary from parseCommonArgs
  • add_fields – List of fields added to the writer not present in the in_file; if None do not add fields
Returns:

Adds a dictionary with key value pairs to collect_queue containing

’log’ defining a log object, ‘out_files’ defining the output file names

Return type:

None

changeo.Multiprocessing.feedDbQueue(alive, data_queue, db_file, group_func=None, group_args={})

Feeds the data queue with Ig records

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues if False exit process
  • data_queue – multiprocessing.Queue to hold data for processing
  • db_file – Database file
  • group_func – Function to use for grouping records
  • group_args – Dictionary of arguments to pass to group_func
Returns:

None

changeo.Multiprocessing.processDbQueue(alive, data_queue, result_queue, process_func, process_args={})

Pulls from data queue, performs calculations, and feeds results queue

Parameters:
  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
  • data_queue – multiprocessing.Queue holding data to process
  • result_queue – multiprocessing.Queue to hold processed results
  • process_func – Function to use for filtering sequences
  • process_args – Dictionary of arguments to pass to process_func
Returns:

None