CARP works by allowing a group of hosts on the same network segment to share an IP address. This group of hosts is referred to as a "redundancy group." The redundancy group is assigned an IP address that is shared amongst the group members. Within the group, one host is designated the "master" and the rest as "backups." The master host is the one that currently "holds" the shared IP; it responds to any traffic or ARP requests directed towards it. Each host may belong to more than one redundancy group at a time.
One common use for CARP is to create a group of redundant firewalls. The virtual IP that is assigned to the redundancy group is configured on client machines as the default gateway. In the event that the master firewall suffers a failure or is taken offline, the IP will move to one of the backup firewalls and service will continue unaffected.
CARP supports IPv4 and IPv6.
It's possible for multiple CARP groups to exist on the same network segment. CARP advertisements contain the Virtual Host ID which allows group members to identify which redundancy group the advertisement belongs to.
In order to prevent a malicious user on the network segment from spoofing CARP advertisements, each group can be configured with a password. Each CARP packet sent to the group is then protected by an SHA1 HMAC.
Since CARP is its own protocol, it should have an explicit pass rule in filter rulesets:
pass out on $carp_dev proto carp
$carp_devshould be the physical interface that CARP is communicating over.
# ifconfig carpN create # ifconfig carpN vhid vhid [pass password] [carpdev carpdev] \ [advbase advbase] [advskew advskew] [state state] [group|-group group] \ ipaddress netmask mask
advbasewhen sending CARP advertisements. By manipulating
advskew, the master CARP host can be chosen. The higher the number, the less preferred the host will be when choosing a master. The default is 0. Acceptable values are from 0 to 254.
carpgroup. Each group has a
carpdemotecounter affecting all carp(4) interfaces belonging to that group. As described below, it can be useful to group certain interfaces together for failover purposes.
Further CARP behavior can be controlled via sysctl(8).
advskewto preempt the master. In addition, this option also enables failing over a group of interfaces together in the event that one interface goes down. If one physical CARP-enabled interface goes down, CARP will increase the demotion counter,
carpdemote, by 1 on interface groups that the carp(4) interface is a member of, in effect causing all group members to fail-over together.
net.inet.carp.preemptis 0 (disabled) by default.
# sysctl net.inet.carp.allow=1 # echo 'net.inet.carp.allow=1' >> /etc/sysctl.conf # ifconfig carp1 create # ifconfig carp1 vhid 1 pass mekmitasdigoat carpdev em0 advskew 100 10.0.0.1 netmask 255.255.255.0This sets up the following:
carp1for virtual host #1, enables a password, sets
em0as the interface belonging to the group, and makes this host a backup due to the
100(assuming of course that the master is set up with an
advskewless than 100). The shared IP assigned to this group is 10.0.0.1/255.255.255.0.
carp1shows the status of the interface.
# ifconfig carp1 carp1: flags=8802<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500 carp: BACKUP carpdev em0 vhid 1 advbase 1 advskew 100 groups: carp inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
When pfsync(4) is set up to send and receive updates on the network, the default behavior is to multicast updates out on the local network. All updates are sent without authentication. Best common practice is either:
syncpeeroption (see below) so that updates are unicast directly to the peer, then configure ipsec(4) between the hosts to secure the pfsync(4) traffic.
pass on $sync_if proto pfsync
$sync_ifshould be the physical interface that pfsync(4) is communicating over.
# ifconfig pfsyncN syncdev syncdev [syncpeer syncpeer] [defer|-defer]
deferflag is used, the initial packet of a new connection passing through the firewall will not be transmitted until either another pfsync(4) system has acknowledged the state table addition, or a timeout has expired. This adds small delays but allows traffic to flow when more than one firewall might actively handle packets ("active/active"), e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations.
# ifconfig pfsync0 syncdev em1 upThis enables pfsync on the
em1interface. Outgoing updates will be multicast on the network allowing any other host running pfsync to receive them.
An example scenario.
The firewalls are connected back-to-back using a crossover cable on+----| WAN/Internet |----+ | | em2| |em2 +-----+ +-----+ | fw1 |-em1----------em1-| fw2 | +-----+ +-----+ em0| |em0 | | ---+-------Shared LAN-------+---
em1. Both are connected to the LAN on
em0and to a WAN/Internet connection on
em2. IP addresses are as follows:
fw1will be the preferred master.
! enable preemption and group interface failover # sysctl net.inet.carp.preempt=1 # echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf ! configure pfsync # ifconfig em1 10.10.10.1 netmask 255.255.255.0 # ifconfig pfsync0 syncdev em1 # ifconfig pfsync0 up ! configure CARP on the LAN side # ifconfig carp1 create # ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \ 172.16.0.100 netmask 255.255.255.0 ! configure CARP on the WAN/Internet side # ifconfig carp2 create # ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \ 192.0.2.100 netmask 255.255.255.0Configure fw2:
! enable preemption and group interface failover # sysctl net.inet.carp.preempt=1 # echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf ! configure pfsync # ifconfig em1 10.10.10.2 netmask 255.255.255.0 # ifconfig pfsync0 syncdev em1 # ifconfig pfsync0 up ! configure CARP on the LAN side # ifconfig carp1 create # ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \ advskew 128 172.16.0.100 netmask 255.255.255.0 ! configure CARP on the WAN/Internet side # ifconfig carp2 create # ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \ advskew 128 192.0.2.100 netmask 255.255.255.0
To failover a particular CARP group, shut down the carp(4) interface on the
This will cause the master to advertise itself with an "infinite"
The backup host(s) will see this and immediately take over the role of master.
# ifconfig carp1 downAn alternative is to increase the
advskewto a value that's higher than the
advskewon the backup host(s). This will cause a failover but still allow the master to participate in the CARP group.
Another method of failover is to tweak the CARP demotion counter. The demotion counter is a measure of how "ready" a host is to become master of a CARP group. For example, while a host is in the middle of booting up it's a bad idea for it to become the CARP master until all interfaces have been configured, all network daemons have been started, etc. Hosts advertising a high demotion value will be less preferred as the master.
A demotion counter is stored in each interface group that the CARP interface belongs to. By default, all CARP interfaces are members of the "carp" interface group. The current value of a demotion counter can be viewed using ifconfig(8):
# ifconfig -g carp carp: carp demote count 0In this example, the counter associated with the "carp" interface group is shown. When a CARP host advertises itself on the network, it takes the sum of the demotion counters for each interface group the carp(4) interface belongs to and advertises that value as its demotion value.
Now assume the following example. Two firewalls running CARP with the following CARP interfaces:
First, assign each to a new interface group, in this case named "internal":
# ifconfig carp1 group internal # ifconfig carp2 group internal # ifconfig internal carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 100 groups: carp internal inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 100 groups: carp internal inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255Now increase the demotion counter for the "internal" group using ifconfig(8):
# ifconfig -g internal internal: carp demote count 0 # ifconfig -g internal carpdemote 50 # ifconfig -g internal internal: carp demote count 50The firewall will now gracefully failover on the carp1 and carp2 groups to the other firewall in the cluster while still remaining the master on carp3 and carp4. If the other firewall started advertising itself with a demotion value higher than 50, or if the other firewall stopped advertising altogether, then this firewall would again take over mastership on carp1 and carp2.
To fail back to the primary firewall, reverse the changes:
# ifconfig -g internal -carpdemote 50 # ifconfig -g internal internal: carp demote count 0Network daemons such as OpenBGPD and sasyncd(8) make use of the demotion counter to ensure that the firewall does not become master until BGP sessions become established and IPsec SAs are synchronized.
carp0). So, write your rule sets accordingly. Don't forget that an interface name in a PF rule can be either the name of a physical interface or an address associated with that interface. For example, this rule could be correct:
pass in on fxp0 inet proto tcp from any to carp0 port 22But replacing the
carp0would not work as you desire.
DON'T forget to pass