changeo.Multiprocessing
Multiprocessing
- class changeo.Multiprocessing.DbData(key, records)
Bases:
object
A class defining data objects for worker processes
- id
result identifier
- data
list of data records
- valid
True if preprocessing was successfull and data should be processed
- class changeo.Multiprocessing.DbResult(key, records)
Bases:
object
A class defining result objects for collector processes
- id
result identifier
- data
list of original data records
- results
list of processed records
- data_pass
list of records that pass filtering for workers that split data before processing
- data_fail
list of records that failed filtering for workers that split data before processing
- valid
True if processing was successful and results should be written
- log
OrderedDict of log items
- property data_count
- changeo.Multiprocessing.collectDbQueue(alive, result_queue, collect_queue, db_file, label, fields, writer=<class 'changeo.IO.AIRRWriter'>, out_file=None, out_args={'failed': False, 'log_file': None, 'out_dir': None, 'out_name': None, 'out_type': 'tsv'})
Pulls from results queue, assembles results and manages log and file IO
- Parameters:
alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns.
result_queue – multiprocessing.Queue holding worker results.
collect_queue – multiprocessing.Queue to store collector return values.
db_file – database file name.
label – task label used to tag the output files.
fields – list of output fields.
writer – writer class.
out_file – output file name. Automatically generated from the input file if None.
out_args – common output argument dictionary from parseCommonArgs.
- Returns:
- Adds a dictionary with key value pairs to collect_queue containing
’log’ defining a log object along with the ‘pass’ and ‘fail’ output file names.
- Return type:
None
- changeo.Multiprocessing.feedDbQueue(alive, data_queue, db_file, reader=<class 'changeo.IO.AIRRReader'>, group_func=None, group_args={})
Feeds the data queue with Ig records
- Parameters:
alive – multiprocessing.Value boolean controlling whether processing continues if False exit process
data_queue – multiprocessing.Queue to hold data for processing
db_file – database file
reader – database reader class
group_func – function to use for grouping records
group_args – dictionary of arguments to pass to group_func
- Returns:
None
- changeo.Multiprocessing.processDbQueue(alive, data_queue, result_queue, process_func, process_args={}, filter_func=None, filter_args={})
Pulls from data queue, performs calculations, and feeds results queue
- Parameters:
alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
data_queue – multiprocessing.Queue holding data to process
result_queue – multiprocessing.Queue to hold processed results
process_func – function to use for processing sequences
process_args – dictionary of arguments to pass to process_func
filter_func – function to use for filtering sequences before processing
filter_args – dictionary of arguments to pass to filter_func
- Returns:
None