217 hash<string, PipelineQueue>
pmap;
Defines an abstract class for accepting data and outputting optionally transformed or filtered data.
Definition: AbstractDataProcessor.qc.dox.h:33
Defines the abstract class for data provider iterators; the destructor releases the iterator.
Definition: AbstractDataProviderBulkRecordInterface.qc.dox.h:33
Defines the abstract class for data provider iterators; the destructor releases the iterator.
Definition: AbstractDataProviderRecordIterator.qc.dox.h:276
Defines a class for passing data through record processors.
Definition: DataProviderPipeline.qc.dox.h:207
string getName()
Returns the pipeline name.
*code error_log
Error log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:246
Counter cnt()
Thread counter.
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline queue threads
Definition: DataProviderPipeline.qc.dox.h:254
int record_count
Record count.
Definition: DataProviderPipeline.qc.dox.h:240
bool hasQueue(int id)
Returns True if the given queue exists, False if not.
bool stop_flag
Stop flag.
Definition: DataProviderPipeline.qc.dox.h:260
hash< string, PipelineQueue > pmap
Hash of queues keyed by queue ID.
Definition: DataProviderPipeline.qc.dox.h:217
hash< PipelineInfo > getInfo()
Returns pipeline info.
abort(*bool ignore_exceptions)
Aborts execution of a pipeline in progress.
Sequence seq(1)
Pipeline ID sequence generator.
submitBulkIntern(AbstractDataProviderBulkRecordInterface i)
Submits bulk data for processing.
logError(string fmt)
Logs to the error log, if set.
copy()
Copy constructor; creates an empty pipeline with the same configuration as the original.
bool stopping()
Returns True if the object is stopping.
registerThread(PipelineQueue queue)
Registers a new thread.
throwPipelineException()
Throws an exception if errors occured in background pipeline processing.
resetIntern()
Resets the pipeline.
submitIntern(auto _data)
Submits data for processing.
list< hash< ExceptionInfo > > error_list
list of exceptions in pipelines
Definition: DataProviderPipeline.qc.dox.h:231
bool aborting()
Returns True if the object is aborting.
bool isProcessing()
Returns True if the pipeline is processing data.
PipelineQueue copyPipeline(PipelineQueue old_queue)
Called by the copy constructor to copy the queues.
int appendQueue(int id)
Appends a new queue to an existing pipeline and returns the new queue ID.
*code debug_log
Debug log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:249
reportError(PipelineQueue queue, hash< ExceptionInfo > ex)
Called from a pipeline queue object to report a fatal error durring processing.
*code info_log
Info log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:243
string name
A descriptive name for logging purposes.
Definition: DataProviderPipeline.qc.dox.h:214
append(int id, AbstractDataProcessor processor)
Appends a data processor to a queue.
date stop_time
run stop time (set in waitDone())
Definition: DataProviderPipeline.qc.dox.h:237
submit(auto _data)
Submits data for processing.
logInfo(string fmt)
Logs to the info log, if set.
submit(AbstractDataProviderBulkRecordInterface i)
Submits data for processing.
waitDone()
Waits for all queues to have processed remaining data.
date start_time
run start time
Definition: DataProviderPipeline.qc.dox.h:234
bool locked
Locked flag.
Definition: DataProviderPipeline.qc.dox.h:225
logDebug(string fmt)
Logs to the debug log, if set.
submit(AbstractDataProviderRecordIterator i)
Submits data for processing.
destructor()
Destroys the object.
submitData(AbstractIterator i)
Submits data for processing.
bool abort_flag
Abort flag.
Definition: DataProviderPipeline.qc.dox.h:263
checkUpdatePipelineIntern(int id)
Check if the given queue exists.
stopInternUnlocked()
Stops all background pipeline queues; lock must be held.
checkSubmitIntern()
Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise.
bool do_bulk
Bulk flag.
Definition: DataProviderPipeline.qc.dox.h:222
checkLockedIntern()
Throws an exception if the pipeline is locked.
append(AbstractDataProcessor processor)
Appends a data processor to the default queue.
reset()
Resets the pipeline.
submitDataIntern(auto _data)
Submits a single record for processing.
constructor(*hash< PipelineOptionInfo > opts)
Creates the object with the given options.
stopIntern()
Stops all background pipeline queues.
Pipeline element.
Definition: DataProviderPipeline.qc.dox.h:110
bool do_flush
Flush pipeline flag.
Definition: DataProviderPipeline.qc.dox.h:154
Mutex lck
Parent lock.
Definition: DataProviderPipeline.qc.dox.h:117
list< auto > queue
Data queue.
Definition: DataProviderPipeline.qc.dox.h:138
int getId()
Returns the pipeline ID.
waitDone()
Wait for the queue to be empty, then wait for all terminating pipelines to be empty.
int data_waiting
Number of threads waiting on data.
Definition: DataProviderPipeline.qc.dox.h:132
list< auto > elems()
Pipeline elements.
int id
Queue ID.
Definition: DataProviderPipeline.qc.dox.h:114
Condition cond()
Queue condition variable.
submit(auto qdata)
Submits data for processing.
Condition flush_cond()
Flush condition variable.
int tid
TID of the background thread.
Definition: DataProviderPipeline.qc.dox.h:144
constructor(DataProviderPipeline parent, Mutex lck, Counter cnt, int id, int size)
Creates the object.
bool data_flushed
Data flushed confirmation.
Definition: DataProviderPipeline.qc.dox.h:157
int size
Maximum queue size.
Definition: DataProviderPipeline.qc.dox.h:141
int queue_waiting
Number of threads waiting data to be removed from the queue.
Definition: DataProviderPipeline.qc.dox.h:129
run(Counter run_cnt)
Processing thread.
Counter cnt
Parent counter.
Definition: DataProviderPipeline.qc.dox.h:120
int flush_waiting
Number of threads waiting on the flush cond.
Definition: DataProviderPipeline.qc.dox.h:135
DataProviderPipeline parent
Parent object.
Definition: DataProviderPipeline.qc.dox.h:160
Qore AbstractDataField class definition.
Definition: AbstractDataField.qc.dox.h:27
const PS_IDLE
Pipeline status: IDLE.
Definition: DataProviderPipeline.qc.dox.h:43
const PS_RUNNING
Pipeline status: RUNNING.
Definition: DataProviderPipeline.qc.dox.h:40
const PS_ABORTED
Pipeline status: ABORTED.
Definition: DataProviderPipeline.qc.dox.h:37
Pipeline info.
Definition: DataProviderPipeline.qc.dox.h:47
date duration
Total time processing end to end.
Definition: DataProviderPipeline.qc.dox.h:74
*date stop_time
Stop time for processing.
Definition: DataProviderPipeline.qc.dox.h:55
float recs_per_sec
Records processed per second end to end.
Definition: DataProviderPipeline.qc.dox.h:80
*date start_time
Start of processing.
Definition: DataProviderPipeline.qc.dox.h:52
bool bulk
Flag that indicates if the pipeline is capable of bulk record processing.
Definition: DataProviderPipeline.qc.dox.h:71
int num_queues
Number of pipeline queues.
Definition: DataProviderPipeline.qc.dox.h:66
int record_count
Number of input records submitted.
Definition: DataProviderPipeline.qc.dox.h:63
string status
Pipeline status.
Definition: DataProviderPipeline.qc.dox.h:60
float duration_secs
Total time processing end to end as a floating-point value in durationSecondsFloat.
Definition: DataProviderPipeline.qc.dox.h:77
string name
The name of the pipeline.
Definition: DataProviderPipeline.qc.dox.h:49
Pipeline option info.
Definition: DataProviderPipeline.qc.dox.h:84
*string name
the name of the pipeline for logging purposes; if this key is not included, a generic name will be ge...
Definition: DataProviderPipeline.qc.dox.h:106
*code info_log
a closure or call reference for info logging
Definition: DataProviderPipeline.qc.dox.h:98
*code error_log
a closure or call reference for error logging
Definition: DataProviderPipeline.qc.dox.h:93
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline queue threads
Definition: DataProviderPipeline.qc.dox.h:103
*code debug_log
a closure or call reference for debug logging
Definition: DataProviderPipeline.qc.dox.h:88