BATCH AND STREAM PROCESS
Batch n stream processing flow: there are window slide, fixed, session (can be selected from
the required use case)
Monitoring to see if there are any problems with the application we are running. If there is a
problem, we have to deal with it. Detect data anomaly errors
We can immediately process and analyze the data that appears (make decisions)
Critical time requires streaming piepline data
Non-analysis data = OLTP [if the storage is heavy, large companies don't mess with OLTP
because it will impact the performance of the application]
Data analysis = OLAP
So, data from OLTP is immediately moved to OLAP so that 1 million users are not disturbed
by slow applications
Streaming is expensive because the service has to run 24 hours. This is different from batches,
for example monthly, so the system runs once a month
If the streaming process time for the e-commeerce promo = so you need to increase the pipeline
specifications
Data pipeline = data flow, the basic task of a data pipeline === moving data from source to
target
So there is a process in the data pipeline
If the batch is charged from a script that we created (we take it from the source, not the source
that was sent to us)
Message system = so that two systems can communicate / activate two-way and asynchronous
communication applications (chat via WA, if the opponent has read it, it has been delivered)
If it's synchronous (call via WA, if you don't pick up, wait for the message, then call again)
There are 2 types =
• Point to point type (point to point, 1 message can only be used by 1 consumer)
There is a queue = queuing system
• Pub-sub
There is a topic
Use case = pen moves data from 1 OLTP to 2 OLAP (1 as primary, 2 as backup) using PUB-
SUB
Source → message system → processing engine → message system → target
