![]() Flume Sink: Generally, to store data into centralized stores like HBase and HDFS we use the Flume sink component.Basically, these channels can work with any number of sources and sinks are they are fully transactional. To be very specific it acts as a bridge between the sources and the sinks in Flume. Flume Channel: A transient store that receives the events from the source also buffers them till they are consumed by sinks is what we call a Flume channel.Moreover, each source receives events from a specified data generator. There are various types of sources Apache Flume supports. Then it transfers it to one or more channels as Flume events. Flume Source: Basically, Flume source receives data from the data generators.Note that, it is possible that Flume can have more than one agent. Afterward, it forwards it to its next destination that is sink or agent. At first, it receives events from clients or other agents. Flume Agents However, in Apache Flume, an independent daemon process (JVM) is what we call an agent.Basically, that can be transported from the source to the destination accompanied by optional headers. Generally, it contains a payload of the byte array. Flume Events The basic unit of the data which is transported inside Flume is what we call an Event.Let’s now talk about each of the components present in the Flume architecture: Then, it is pushed to a centralized store, i.e., HDFS. The data collector is another agent that collects data from various other agents that are aggregated. This data that has been generated gets collected by Flume agents. In Flume architecture, there are data generators that generate data. Refer to the image below for understanding the Flume architecture better. Data is carried between sources and sinks by Apache Flume which can either be event-driven or can be scheduled.This design stands out to be robust and fault-tolerant with different recovery mechanisms. Based on streaming data flows, Flume has a flexible design. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |