Summary -
In this topic, we described about the below sections -
The Channel selector is that component of Flume that determines which channel particular Flume event should go into when a group of channels exists. The target channel can be one or multiple.
The mechanism used is an internal mechanism. As discussed earlier, in two ways the multiple channels can be handled. There are different types of channel selectors.
Replicating channel selectors -
This is the default channel selector. If anything do not configured with respect to the channel selector, then it is the replicating channel selector that will do the job of deciding the channel.
It simply puts a copy of the event into each channel assuming more than one channel exists.
# Replicating channel selectors
<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.type = replicating
<Agent_name>.sources.<source-name>.channels
= <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.optional
= <Optional channel-number>
Example for agent named agt1 and it’s source called src1 -
agt1.sources = src1
agt1.channels = chnl1 chnl2 chnl3
agt1.sources.src1.selector.type = replicating
agt1.sources.src1.channels = chnl1 chnl2 chnl3
agt1.sources.src1.selector.optional = chnl1
In the above configuration, chnl1 is an optional channel. Chnl1 failure to write to is simply ignored. Chnl2 and chnl3 failure to write channels will cause the transaction to fail as these channels marked as not optional.
Below are the two properties for replicating channel selector.
| Property Name | Default | Description |
|---|---|---|
| selector.type | replicating | The component type name, needs to be replicating |
| selector.optional | – | Set of channels to be marked as optional |
Multiplexing channel selector -
This channel selector can write the Flume event to different channels depending on the header information.
# Multiplexing channel selectors
<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.type = multiplexing
<Agent_name>.sources.<source-name>.selector.header = <header-name>
<Agent_name>.sources.<source-name>.selector.mapping.
<header-category> = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.mapping.<header-category>
= <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.default
= <Channel1> <Channel2>……<Channeln>
Example for agent named agt1 and source called src1 -
agt1.sources = src1
agt1.channels = chnl1 chnl2 chnl3 chnl4
agt1.sources.src1.selector.type = multiplexing
agt1.sources.src1.selector.header = grade
agt1.sources.src1.selector.mapping.grade1 = chnl1
agt1.sources.src1.selector.mapping.grade2 = chnl2
agt1.sources.src1.selector.mapping.grade3 = chnl3
agt1.sources.src1.selector.default = chnl4
In the above configuration, channels are separated by the header grade. Grade1 are routed to chnl1, grade2 are routed to chnl2 and grade3 are routed to chnl3. chnl4 is an default channel.
Below are the two properties for multiplexing channel selector.
| Property Name | Default | Description |
|---|---|---|
| selector.type | replicating | The component type name, needs to be multiplexing |
| selector.header | flume.selector.header | |
| selector.default | – | |
| selector.mapping.* | – |
Custom channel selector -
This is users own implementation of the ChannelSelector interface. A custom channel selector’s class and its dependencies must be included in the agent’s classpath. The Flume starign should be done after custom channel selector included.
# custom channel selectors
<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1>
<Agent_name>.sources.<source-name>.selector.type
= custom selector type
Example for agent named agt1 and its source called src1 -
agt1.sources = src1
agt1.channels = chnl
agt1.sources.src1.selector.type = example.ChannelSelector
Below are the two properties for custom channel selector.
| Property Name | Default | Description |
|---|---|---|
| selector.type | – | The component type name, needs to be your FQCN |
The selector for any channel can be specified using the selector.type property. The channel selectors work in between the Source and Channel. The channel selector will decide to which channel to which Sink to which HDFS cluster or HBase system.