changeo.Multiprocessing¶
Multiprocessing
-
class
changeo.Multiprocessing.
DbData
(key, records)¶ Bases:
object
A class defining data objects for worker processes
-
id
¶ result identifier
-
data
¶ list of data records
-
valid
¶ True if preprocessing was successfull and data should be processed
-
-
class
changeo.Multiprocessing.
DbResult
(key, records)¶ Bases:
object
A class defining result objects for collector processes
-
id
¶ result identifier
-
data
¶ list of original data records
-
results
¶ list of processed records
-
data_pass
¶ list of records that pass filtering for workers that split data before processing
-
data_fail
¶ list of records that failed filtering for workers that split data before processing
-
valid
¶ True if processing was successful and results should be written
-
log
¶ OrderedDict of log items
-
data_count
¶
-
-
changeo.Multiprocessing.
collectDbQueue
(alive, result_queue, collect_queue, db_file, label, fields, writer=<class 'changeo.IO.ChangeoWriter'>, out_file=None, out_args={'failed': False, 'log_file': None, 'out_dir': None, 'out_name': None, 'out_type': 'tab'})¶ Pulls from results queue, assembles results and manages log and file IO
Parameters: - alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns.
- result_queue – multiprocessing.Queue holding worker results.
- collect_queue – multiprocessing.Queue to store collector return values.
- db_file – database file name.
- label – task label used to tag the output files.
- fields – list of output fields.
- writer – writer class.
- out_file – output file name. Automatically generated from the input file if None.
- out_args – common output argument dictionary from parseCommonArgs.
Returns: - Adds a dictionary with key value pairs to collect_queue containing
’log’ defining a log object along with the ‘pass’ and ‘fail’ output file names.
Return type:
-
changeo.Multiprocessing.
feedDbQueue
(alive, data_queue, db_file, reader=<class 'changeo.IO.ChangeoReader'>, group_func=None, group_args={})¶ Feeds the data queue with Ig records
Parameters: - alive – multiprocessing.Value boolean controlling whether processing continues if False exit process
- data_queue – multiprocessing.Queue to hold data for processing
- db_file – database file
- reader – database reader class
- group_func – function to use for grouping records
- group_args – dictionary of arguments to pass to group_func
Returns: None
-
changeo.Multiprocessing.
processDbQueue
(alive, data_queue, result_queue, process_func, process_args={}, filter_func=None, filter_args={})¶ Pulls from data queue, performs calculations, and feeds results queue
Parameters: - alive – multiprocessing.Value boolean controlling whether processing continues; when False function returns
- data_queue – multiprocessing.Queue holding data to process
- result_queue – multiprocessing.Queue to hold processed results
- process_func – function to use for processing sequences
- process_args – dictionary of arguments to pass to process_func
- filter_func – function to use for filtering sequences before processing
- filter_args – dictionary of arguments to pass to filter_func
Returns: None