changeo.Multiprocessing

Multiprocessing

class changeo.Multiprocessing.DbData(key, records)

Bases: object

A class defining data objects for worker processes

id

result identifier

data

list of data records

valid

True if preprocessing was successfull and data should be processed

class changeo.Multiprocessing.DbResult(key, records)

Bases: object

A class defining result objects for collector processes

id

result identifier

data

list of original data records

results

list of processed records

data_pass

list of records that pass filtering for workers that split data before processing

data_fail

list of records that failed filtering for workers that split data before processing

valid

True if processing was successful and results should be written

log

OrderedDict of log items

property data_count
changeo.Multiprocessing.collectDbQueue(alive, result_queue, collect_queue, db_file, label, fields, writer=<class 'changeo.IO.AIRRWriter'>, out_file=None, out_args={'failed': False, 'log_file': None, 'out_dir': None, 'out_name': None, 'out_type': 'tsv'})

Pulls from results queue, assembles results and manages log and file IO

Parameters

  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns.

  • result_queue – multiprocessing.Queue holding worker results.

  • collect_queue – multiprocessing.Queue to store collector return values.

  • db_file – database file name.

  • label – task label used to tag the output files.

  • fields – list of output fields.

  • writer – writer class.

  • out_file – output file name. Automatically generated from the input file if None.

  • out_args – common output argument dictionary from parseCommonArgs.

Returns

Adds a dictionary with key value pairs to collect_queue containing

’log’ defining a log object along with the ‘pass’ and ‘fail’ output file names.

Return type

None

changeo.Multiprocessing.feedDbQueue(alive, data_queue, db_file, reader=<class 'changeo.IO.AIRRReader'>, group_func=None, group_args={})

Feeds the data queue with Ig records

Parameters

  • alive – multiprocessing.Value boolean controlling whether processing continues if False exit process

  • data_queue – multiprocessing.Queue to hold data for processing

  • db_file – database file

  • reader – database reader class

  • group_func – function to use for grouping records

  • group_args – dictionary of arguments to pass to group_func

Returns

None

changeo.Multiprocessing.processDbQueue(alive, data_queue, result_queue, process_func, process_args={}, filter_func=None, filter_args={})

Pulls from data queue, performs calculations, and feeds results queue

Parameters

  • alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns

  • data_queue – multiprocessing.Queue holding data to process

  • result_queue – multiprocessing.Queue to hold processed results

  • process_func – function to use for processing sequences

  • process_args – dictionary of arguments to pass to process_func

  • filter_func – function to use for filtering sequences before processing

  • filter_args – dictionary of arguments to pass to filter_func

Returns

None