Network Management with the OpenBSD Packet Filter Toolset BSDCan 2023

# Network Management with the OpenBSD Packet Filter Toolset

## BSDCan 2023

#### Ottawa, May 18th, 2023

### Peter Hansteen, Massimiliano Stucchi, Tom Smyth

---

## whoami (who am I), part Peter:

### Peter Hansteen <peter@bsdly.net>

- Sysadmin, [OpenBSD](https://www.openbsd.org) user since before the millenium 
	
- Wrote [The Book of PF](https://www.nostarch.com/pf3), now in its third edition

- Blog at [bsdly.blogspot.com](https://bsdly.blogspot.com) about (lack of) sanity in IT

- Works at [Tietoevry Create](https://tietoevry.com/en/create)

- Yes, I'll do another book any decade now

---

## whoami (who am I), part Max:

### Massimiliano Stucchi

- Technical Advisor at The Internet Society
    - Here representing myself only

- IPv6 "Enthusiast"

- https://stucchi.ch
  - @stucchimax@social.secret-wg.org

---
## whoami (who am I), part Tom:

### Tom Smyth

- working in IT since 2000
  - CTO wireless Connect Ltd. an ISP in Ireland
  - Opinions are mine and may be my companies also :)
  - PF student, an avid reader of the Book of PF.
  - I really Enjoy networking with OpenBSD
  - Maintainer of the NSH network Shell for OpenBSD.

---

## Introduce yourself

- A quick introduction about yourself:

- Your name

- Your favourite BSD

- Your experience with networking

- Your experience with PF

- Your goal(s)

???

This introduction serves for us to understand the level of the room, and decide on how to better suit
the tutorial.

Getting an idea of the level makes it so that we might quickly go through the basics, and focus on something
more advanced, rather than spend time on something that everybody knows already.

---

## Agenda

1. PF Basics
2. Exercise: Host configuration
3. NAT and Redirects
4. Exercise: Setup a gateway
5. Hosting Services
6. Exercise: Hosting Services, redirects
7. Traffic Shaping
8. Exercise: Setting up queueing
9. Tips
10. Troubleshooting
11. Exercise: NAT64
12. End

---

# PF Basics
  
  
### Section 1

---
## It all started with a copyright violation

- OpenBSD up to 2.9 (2001) used Darren Reed's IPFilter

- Almost, but not quite BSD licensed

- No right to distribute changed versions

- IPFilter removed on May 29th, 2001

- First commit of the new PF code June 24 2001 at 19:48:58 UTC

- OpenBSD 3.0 release pushed to Dec 1 by the extra effort

- *License audit* of src tree + ports tree followed

But let's jump to the present -

???

---
## Network design 101

How to build a maintainable network:

- Start with something simple

- Think dual stack (IPv6 and IPv4)

- Users
  - Access to local and remote services
    - and their security requirements

- The services you offer and the ones you consume
  - i.e: SMTP, WWW, DNS

- **Keep it simple, not stupid** 
---
## Know what a Firewall can (and cannot) do!

A firewall can:
- allow certain packets flowing to or through the firewall device
- drop certain packets flowing to or through the firewall device
- redirect or NAT packets according to policy

There are limits though
- Inbound interface bandwith limitations
- CPU I/O interface driver pacekt per second(PPS) limitations

A firewall cannot
- block inbound Flood Denial of Service attacks that exceed:
  - the inbound interface capacity 
  - the CPU I/O / interupt capacity of the interface

Any software firewall can be overwhelmed with high packet rate Denial of Service attacks.
If the scale of the attack exceeds the capability of the firewall

---
## But first, how do we build rulesets?

The basic _rule_ format is,

**verb** _criteria_ **actions** ... options

```shell
 pass in on egress proto tcp to egress port ssh
```

```shell
 match out on egress nat-to egress
```

```shell
 block all
```

You can do complete rulesets with only these, but you may also want to declare things ->

( **egress** ? -> the _interface group_ consisting of interfaces that have a default route )

---
## Next, declarations

Declaring objects your rules will use:

**Macros:**

```shell
macro-name = "definition"
```

**Tables:**

```shell
table <tablename> 
```

**Queues:**

```shell
queue queuename on interface bandwidth number ...
```

You can also *set* options.

We'll be returning to all of those, in the meantime [man pf.conf](http://man.openbsd.org/pf.conf) is always your friend.

---
## Block by default

- A firewall is supposed to block by default, right?

- Let's start with the ruleset for a client

- Useful baseline:
  - block by default
  - pass traffic we generate

```shell
    block
    pass from (self)
```

---
## In practice

- On my laptop in my home network, that expands to:

```shell
    $ doas pfctl -vnf /etc/pf.conf
    block drop all
    pass inet6 from ::1 to any flags S/SA
    pass on lo0 inet6 from fe80::1 to any flags S/SA
    pass on iwm0 inet6 from fe80::a2a8:cdff:fe63:abb9 to any flags S/SA
    pass inet6 from 2001:470:28:658:a2a8:cdff:fe63:abb9 to any flags S/SA
    pass inet6 from 2001:470:28:658:8c43:4c81:e110:9d83 to any flags S/SA
    pass inet from 127.0.0.1 to any flags S/SA
    pass inet from 192.168.103.126 to any flags S/SA
```
???

It would be nice in this slide to ask questions, to see the general knowledge about protocols:

- IPv6 LLA
- IPv6 GUA
- IPv6 GUA with privacy extensions
- IPv4 Private address

This can give us an idea of how to proceed with IPv6 depending on the level of the participants.

---
##   Anatomy of a ruleset

- Ruleset evaluated top to bottom

- **Last match wins**  (unless _quick_ is used)

- Stateful by default

- For more details, see [man pf.conf](http://man.openbsd.org/pf.conf)
  - http://man.openbsd.org/pf.conf

---
## A ruleset can contain

- **Options**
	- General configuration

- **Macros**
	- Content to be expanded
  
- **Tables**
	- When macros fall short (for IP addresses only)

- **Redirections**

- **Rules**
	- How PF should behave

- The order of the items in the ruleset used to be important, but not anymore

---

## Options

- General configuration for pf

- Useful for debugging, applying default timeout values, etc.

```shell
    set limit states 100000
```

- or

```shell
    set debug debug
    set loginterface dc0
    set timeout tcp.first 120 
    set timeout tcp.established 86400 
    set timeout { adaptive.start 6000, adaptive.end 12000 }
```

---

## Macros

- Content that will later be expanded in context

- Name must start with a _letter_, _digit_, or _underscore_

- Useful to keep rulesets simple and clean

```shell
    ext_if = "kue0" 
    all_ifs = "{" $ext_if lo0 "}" 
    pass out on $ext_if from any to any 
    pass in  on $ext_if proto tcp from any to any port 25
```

- Could also be port or address ranges

```shell
   bacula_ports = "9101:9103"
   dmz_hosts = "192.0.2.1 - 192.0.2.254"
```

???

If we have more slide space, then separate the second example in its own slide

---
##   Tables

- When IP address list macros grow too long, use tables instead

- Hold only _addresses_ and _networks_
  - No port ranges

- Provide fast lookups
 
```shell
 table <bruteforce> persist counters
```

- They can be used in rules

```shell
 block from <bruteforce>
```

- Unlike macros with port or address lists, _do not_ expand to multiple rules
  - More optimised ruleset

---

## Tables continued

- Content can be loaded from files or from CLI

```shell
 table <popflooders> persist counters file "/home/peter/popflooders"
```

useful if you save contents to preserve across reboots, such as
```shell
   $ doas pfctl -t popflooders -T show >/home/peter/popflooders
```
possibly in [cron](https://man.openbsd.org/cron) jobs

- use [pfctl](http://man.openbsd.org/pfctl) to add/remove entries from command line

```shell
   $ doas pfctl -t bruteforce -T add 192.0.2.11 2001:db8::dead:beef:baad:f00d
```
  
- optionally maintained by daemons such as [spamd](http://man.openbsd.org/spamd), [bgpd](http://man.openbsd.org/bgpd) and [dhcpd](http://man.openbsd.org/dhcpd)

---

##   Implementing your spec

### How would you do this in your home/soho infrastructure ?

???

We ask the participants how they think they would implement this in their infrastructure.

This is to create discussion, and hopefully make it so that we all learn a bit by trial-and-error.

Let's put out the question, wait for some time to see if anyone comes up with a nice description, and then walk them through what is a sensible setup.

This is in preparation for the first exercise which comes next.

---
class: center, middle

# Exercise 1

### Protecting your host

---
## Excercise 1 - Let's start

- Lab environment:
  - Open your favourite browser, then
  - Go to [labs.pftutorial.net](https://labs.pftutorial.net)
  - Username is "labX" (where X is your assigned number)
  - Password is "pftutorial"
  
---
## Exercise 1 - First steps

- Get to the console for __fwlabX__

- Check that pf is indeed loaded and running (*hint*: [pfctl](http://man.openbsd.org/pfctl))

- Wait until everyone has connected in order to proceed

---

## Exercise 1 - net config

- Configure the external interface on gateway

- _vi /etc/hostname.vio0_

```shell 
		inet 10.255.255.XX/24
		!route add 0/0 10.255.255.254
		inet6 fd18:b5d:cafe::XX/64
		!route add -inet6 2000::/3 fd18:b5d:cafe::a
		!route add -inet6 fd00::/8 fd18:b5d:cafe::a
```

- and then _vi /etc/resolv.conf_

```shell 
    nameserver 10.255.255.254
    nameserver fd18:b5d:cafe::a
```

followed by

```shell 
    sh /etc/netstart
```

---

## Exercise 1 - on gateway

- Start with a block ruleset

```shell 
    block
    pass quick inet6 proto tcp from fd18::/16 to port ssh
		pass quick inet6 proto icmp6 from fd18::/16
```

- Allow traffic to be generated from your host, and allow ICMPv6

```shell
	pass from self
```

and then, reload _pf.conf_

```shell
   pfctl -vnf /etc/pf.conf
   pfctl -f /etc/pf.conf
```

- **NB:** Reload pf this way after every statement in the exercises

---

## Exercise 1 - Tests

- From your gateway ping a host

- First IPv6

```shell 
 # ping6 fd18:b5d:cafe::a
 PING fd18:b5d:cafe::a (fd18:b5d:cafe::a): 56 data bytes
 64 bytes from fd18:b5d:cafe::a: icmp_seq=0 hlim=64 time=0.548 ms
 64 bytes from fd18:b5d:cafe::a: icmp_seq=1 hlim=64 time=0.492 ms
 64 bytes from fd18:b5d:cafe::a: icmp_seq=2 hlim=64 time=0.494 ms

```

- Then IPv4

```shell 
 # ping stucchi.ch
 PING stucchi.ch (45.129.224.40): 56 data bytes
 64 bytes from 45.129.224.40: icmp_seq=0 ttl=56 time=6.264 ms
 64 bytes from 45.129.224.40: icmp_seq=1 ttl=56 time=6.273 ms
 64 bytes from 45.129.224.40: icmp_seq=2 ttl=56 time=6.117 ms
```

---

## Exercise 1 - Wrap up

- Does ping work?

- Do other commands work?
  - working from total block, proceed to make restricted workstation
    - name resolution
    - _http_ and _https_

- Access public web sites, other Internet resources.

- What would it take to access the other lab hosts?

---

# Questions ?

???

Let's ask if there are any questions before continuing.  Make sure we have everyone onboard.

---

# Gateway, NAT and Redirects 
  
  
### Section 2

---
## Easy! How do we improve on this?

- Make a 'firewall':

- a point of policy enforcement

- a gateway

- filter for other hosts

- redirection tricks

---
## Introducing NAT

- **N**etwork **A**ddress **T**ranslation ([RFC1631](https://tools.ietf.org/html/rfc1631) onwards)

- -> 'Hide' several hosts behind 1 or more public addresses, using [RFC1918](https://tools.ietf.org/html/rfc1918) addresses
- -> can be used by ISPs for conserving scarce IP addresses in large networks (CG-NAT) 100.64.0.0/10

- Modern PF has _nat-to_ on 'pass' and 'match' rules:

```shell
	match out on $extif inet nat-to ($extif)
```

- *Neat trick*: egress is the interface group that has a default route, you can filter on it

```shell
	match out on egress inet nat-to (egress)
```

- In modern networks we **should** (also) have IPv6 (inet6)

???

NAT, the stopgap measure that's old enough to drink, more than 22 years old.  NAT was created as a temporary measure that hasn't been replaced by now.  Not even IPv6.

We haven't discusses "egress" yet, so this is the right time to introduce it.

Spend some time discussing it, along with the rest of the NAT specifications.

Unfortunately, there's also NAT for IPv6, called NAT66

---
## A (filtering) Gateway

_"I decide which packets pass"_

#### Enable forwarding:

- Temporarily set from command line with [sysctl](http://man.openbsd.org/sysctl):

```shell 
    # sysctl net.inet.ip.forwarding=1 
    # sysctl net.inet6.ip6.forwarding=1 
```

- Make permanent in [/etc/sysctl.conf](http://man.openbsd.org/sysctl.conf)

```shell 
    net.inet.ip.forwarding=1
    net.inet6.ip6.forwarding=1
```

---
## The minimal gateway

- Do you *NAT* for IPv4? Of course you do.

- Do you run IPv6? Of course you do.

```shell 
    ext_if=bge0
    int_if=bge1
    match out on egress inet nat-to ($ext_if)
    block all
    pass proto tcp from { self, $int_if:network }
```

- The "pass" rule, withouth _inet_ or _inet6_ applies to both

**Keep in mind**: This is a point of policy enforcement

---
## A Point of policy enforcement

- Now some policy, and macros

```shell 
  ext_if=bge0
  int_if=bge1

client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \
      https, 2628, 5999, 8000, 8080 }
  udp_services = "{ domain, ntp }"

match out on egress inet nat-to ($ext_if)
* block
* pass quick proto { tcp, udp } to port $udp_services keep state
*
* pass proto tcp from $int_if:network to port $client_out
*
* pass proto tcp to self port ssh
```

- What services do your clients consume?

???

Log to on the system we have for showing out, and then show the rules there and how they expand to different parts.

---
## Letting dhcpd(8) direct access

OpenBSD [dhcpd(8)](http://man.openbsd.org/dhcpd) can interact with your ruleset via tables:

*/etc/rc.conf.local*
```shell 
dhcpd_flags="-L leased_ip_table -A abandoned_ip_table -C changed_ip_table bge1"
```

```shell 
 ext_if=bge0
 int_if=bge1
* table <abandoned_ip_table> persist counters
* table <changed_ip_table> persist counters
* table <leased_ip_table> persist counters

client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \
                  https, 2628, 5999, 8000, 8080 }"
  udp_services = "{ domain, ntp }"

match out on egress inet nat-to ($ext_if)
 block
 pass quick proto { tcp, udp } to port $udp_services keep state
* pass proto tcp from <leased_ip_table> to port $client_out
 pass proto tcp to self port ssh
```
**=>** only pass traffic from hosts with active leases from *me*

???

Maybe move this slide to a later section.

---
## Redirects (and divert-to)

Modern PF has two classes of redirect

* **rdr-to** on match and pass rules - rewrite destination address while filtering (locally or even to other hosts)

```shell 
      pass in on egress to port www rdr-to $webserver
```

* **divert-to** on match and pass rules - [divert()](http://man.openbsd.org/divert) socket for local use

```shell 
      pass in on egress to port smtp divert-to 127.0.0.1 port spamd
```

---

## FTP Proxy

- If your users need to access FTP services, [ftp-proxy](http://man.openbsd.org/ftp-proxy) is what you need

- FTP does not easily pass through a block firewall, some help is needed

```shell 
$ doas rcctl enable ftpproxy6
```

- or for IPv4

```shell 
$ doas rcctl enable ftpproxy
```

- and then add an anchor and divert rules to your config

```shell 
anchor "ftp-proxy/*"
  ... 
pass in quick inet proto  tcp to port ftp divert-to 127.0.0.1 port 8021
pass in quick inet6 proto tcp to port ftp divert-to ::1 port 8021
pass out proto tcp from $proxy to port ftp
```

There is even a reverse mode (**-R**) for when you host FTP servers, see [man ftp-proxy](http://man.openbsd.org/ftp-proxy)

---

# Exercise 2
  
  
### Protecting your network

---

## Exercise 2 - Goals

- Your network grows, you become a gateway

- Extend the configuration to enable the network to access the internet

---
background-image: url(images/exercise2.png)

## Exercise 2 - Your network

---

## Exercise 2

- Turn on ip forwarding (sysctl)

```shell 
    # sysctl net.inet.ip.forwarding=1 
    # sysctl net.inet6.ip6.forwarding=1 
```

- Set up NAT

```shell
	match out on egress inet nat-to (egress)
```

Also, pass traffic from that local net

---
## Exercise 2 - preparation

- Configure the hosts with the following IPv6 addresses
 - *Gateway (vio1):* fd18:b5d:XX::a/64
 - *Host1:* fd18:b5d:XX::80/64
 - *Host2:* fd18:b5d:XX::25/64

- On Host1 and Host2, set fd18:b5d:XX::a as the default IPv6 gateway

- and also the following IPv4 addresses
  - *Gateway (vio1):* 192.168.XX.1/24
	- *Host1:* 192.168.XX.2/24
	- *Host2:* 192.168.XX.3/24

- On Host1 and Host2 set 192.168.XX.1 as the default IPv4 gateway

---

## Exercise 2 - check your results

- From client 1, ping a host on the internet

- First IPv6

```shell 
 # ping6 stucchi.ch
 PING stucchi.ch (2001:41d0:8:6ed8::80): 56 data bytes
 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=1 hlim=56 time=7.414 ms
 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=2 hlim=56 time=6.333 ms
 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=3 hlim=56 time=6.441 ms
```

- Then IPv4

```shell 
 # ping stucchi.ch
 PING stucchi.ch (37.59.51.141): 56 data bytes
 64 bytes from 37.59.51.141: icmp_seq=0 ttl=56 time=6.264 ms
 64 bytes from 37.59.51.141: icmp_seq=1 ttl=56 time=6.273 ms
 64 bytes from 37.59.51.141: icmp_seq=2 ttl=56 time=6.117 ms
```

---
# Exercise 2b: FTP

Try fetching *ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest*

```shell
 # wget ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
```

Check your result

If it didn't work, configure FTP-proxy and try again.

---

# Hosting Services
  
  
### Section 3

---

## Hosting services behind your gateway

- Now you actually want some 'pass in on egress' rules :)

- Get your specifications clear, put in writing:

- your services and the ports they use
    - the names could be in **/etc/services** already

- the hosts (IP addresses) that run the services

- decide who/what/where to be reachable for/from

- Check services requiring extra help (i.e. proxying).

---
## Proxies and other helpers

- OpenBSD comes with several proxies and other service helper programs in the base system:

- [ftp-proxy](http://man.openbsd.org/ftp-proxy) (you guessed it)

- [relayd](http://man.openbsd.org/relayd) (load balancing and lots more)

- [spamd](http://man.openbsd.org/spamd) (if you run SMTP - annoy spammers)

- There are also such things as squid and varnish (web proxies) in packages

- Clients and services in the same subnet? Or do the DMZs?
  
---
## DMZ, defined

### '*D*e-*M*ilitarized *Z*one'

- when a group of host needs special treatment

- attached to separate interfaces(s), separate sub-rulesets

- And **YES**, you can have several

- Think multiple customers, *N* environments each

---
background-image: url(images/exercise3.png)
class: right
## DMZ, illustrated

---
##Allowing some services in

```shell 
    ext_if=bge0 # adjust to what your system has
    int_if=bge1 # adjust to what your system has

client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \
                https, 2628, 5999, 8000, 8080 }"
    udp_services = "{ domain, ntp }"

*   webserver = "192.0.2.227"
*   webports = "{ http, https }"
*   emailserver = "192.0.2.225"
*   email = "{ smtp, pop3, imap, imap3, imaps, pop3s }"
*   nameservers = "{ 192.0.2.221, 192.0.2.223 }"

match out on egress inet nat-to ($ext_if)
    block
    pass quick proto { tcp, udp } to port $udp_services keep state
    pass proto tcp from $int_if:network to port $client_out
    pass proto tcp to self port ssh

*   pass proto tcp to $webserver port $webports 
*   pass proto tcp to $emailserver port $email 
*   pass log proto tcp from $emailserver to port smtp 
*   pass inet proto { tcp, udp } to $nameservers port domain

```

Now try loading this with *pfctl -vnf /etc/pf.conf* and see what this expands to (you can also fetch [pf.services01.conf](samples/pf.services01.conf) and try)

---
## Tackling noise (attacks) with state-tracking options

### Scenario: ssh bruteforcers

In our previous ruleset, add

```shell 
table <bruteforce> persist counters
block from <bruteforce>
```

and change the **ssh** rule to

```shell 
* pass proto tcp to port ssh flags S/SA keep state \
* (max-src-conn 15, max-src-conn-rate 2/10, overload <bruteforce> flush global)

```

- Tune to taste
- More info on options in [man pf.conf](http://man.openbsd.org/pf.conf)
- Remember *[pfctl](http://man.openbsd.org/pfctl) expire*

???
max-src-conn - see man page but max number of simultaneous connections from one host
max-src-conn-rate - number of new connections per number of seconds

---

## Tackling noise (attacks) with state-tracking options

### Scenario: Wordpress site

- Wordpress is a usual target for many different attacks

- -> Make the attackers suffer by forcing them through smaller "windows"
  - Use tables and block by connection rates

- Mix and match settings, consult [man pf.conf](http://man.openbsd.org/pf.conf) and remember *[pfctl](http://man.openbsd.org/pfctl) expire*

- Also see [Forcing the password gropers through a smaller hole with OpenBSD's PF queues](http://bsdly.blogspot.ca/2017/04/forcing-password-gropers-through.html), [Badness, Enumerated by Robots](https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html) (blog posts) + [The Book of PF](https://www.nostarch.com/pf3)

---

## Scenario: Wordpress site

- You can re-use the *bruteforce* table or make a separate one, like

```shell 
 table <web_brutes> persist counters
 block from <web_brutes>
```

- Now change the **$webports** rule to

```shell 
* pass proto tcp to port $webports flags S/SA keep state \
* (max-src-conn 15, max-src-conn-rate 5/10, overload <web_brutes> flush global)

```

- Alternatively, on a machine (the gateway) that does not run Wordpress, do

```shell 
  grep wp-login /var/www/logs/access.log | awk '{print $1}' | \ 
    sort -u | xargs doas pfctl -t web_brutes -T add
```

- (**Wordpressers**: Do these look right? Consult web logs and [tcpdump](http://man.openbsd.org/tcpdump) output.)

???
max-src-conn - see man page but max number of simultaneous connections from one host
max-src-conn-rate - number of new connections per number of seconds 
You could also look into max-src-nodes, max-src-states

---

## Annoying spammers with spamd

- [spamd(8)](http://man.openbsd.org/spamd) is good, clean, fun
 
- Speaks enough SMTP to do [greylisting](https://en.wikipedia.org/wiki/Greylisting)
 
- Can tarpit known bad senders and generate blacklists by greytrapping
 
- Default [spamd.conf](http://man.openbsd.org/spamd.conf) gives you one blacklist import and basics
 - (**Hint:** no real SMTP service required)
 
- You can even generate your own blacklists by *greytrapping* via *non-deliverable* spamtrap addresses in your own domain(s)
 
- Also see: 
 - [The Book of PF](https://www.nostarch.com/pf3)
 - [In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes](http://bsdly.blogspot.ca/2012/05/in-name-of-sane-email-setting-up-spamd.html)
 - [Maintaining A Publicly Available Blacklist](http://bsdly.blogspot.ca/2013/04/maintaining-publicly-available.html)

---
## Annoying spammers with spamd

- Set up tables
- Divert traffic to the spamd process
- Only let the "good guys" pass to the real SMTP daemon

```shell 
table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"
pass in on egress proto tcp to any port smtp divert-to 127.0.0.1 port spamd
pass in on egress proto tcp from <nospamd> to any port smtp
pass in log on egress proto tcp from <spamd-white> to any port smtp
pass out log on egress proto tcp to any port smtp
```

- and then run

```shell
$ doas rcctl enable spamd
$ doas rcctl start spamd
```

**TIP:** check out [smtpctl spf walk <_nospamd_domains.txt_](https://man.openbsd.org/smtpctl) to feed your _nospamd_ table from live [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) data (in OpenBSD 6.3 onwards), see the blog post [Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting](https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html)

---
## Scenario: SYN flood vs syncookies

A common Denial-of-Service (DOS) technique is to send large numbers of SYNs from spoofed addresses, filling up the state table.

In OpenBSD 6.3 and newer we have the [pf.conf](http://man.openbsd.org/pf.conf) option

```shell 
set syncookies
```

The default is off, if you enable with

```shell 
set syncookies on 
```

**all** SYNs get SYNACK answer, but no resources allocated until ACK received (pending match to **pass** rule)

The other option is **adaptive** with syncookies used only when half open TCP connections reach the **start** percentage, until the **end** level is reached

```shell 
set syncookies adaptive (start 29%, end 15%) 
```

---
## Consider: what about that separate DMZ?

- What do you need?
 
- IP addresses: Segment off or allocate separate address ranges
 
- Attach each segment to separate interface, VLAN
 
- Do the ruleset surgery
 - which services do you run, where
 - do you need renumbering?
 - what traffic (in and out) is actually required?

???

---

# Questions ?
  
???

Let's ask if there are any questions before continuing.  Make sure we have everyone onboard.

---

### Offering services

---

## Excercise 3 - Goals

- You're now offering services
 
- **Host 1** will provide **http** service
 
- **Host 2** will provide **smtp** service
 
- We need to setup:
 - The services
 - Redirects
 - Firewall rules

---
background-image: url(images/exercise3.png)
class: right
## Exercise 3 - Network

???

This is not exactly the network we have, but could be thought as such.

Host1 and Host2 are in what could be considered our DMZ.

---
## Exercise 3 - on Host1

- We need to configure and start httpd

```shell
 # cp /etc/examples/httpd.conf /etc/httpd.conf

< comment out the HTTPS part >

# rcctl enable httpd
 # rcctl start httpd
 httpd(ok)
```

---
## Exercise 3 - on Host2

- Change the config to listen on all interfaces:
  - Change the appropriate line in
  
### /etc/mail/smtpd.conf

```shell
 listen on all
```
 
- Then start the daemon

```shell
 # rcctl enable smtpd
 # rcctl start smtpd
 smtpd(ok)
```
- (It might take a while)

---
## Exercise 3 - on gateway

### /etc/pf.conf

```shell
webserver_v4 = "$IP_addr_of_host1"
webserver_v6 = "fd18:b5d:XX::80"
webports = "{ http, https }"
emailserver_v4 = "$IP_addr_of_host2"
emailserver_v6 = "fd18:b5d:XX::25"
email = "{ smtp, pop3, imap, imap3, imaps, pop3s }"

match in on egress inet proto tcp to egress port $webports rdr-to $webserver_v4
match in on egress inet proto tcp to egress port $email rdr-to $emailserver_v4

pass inet proto tcp to $webserver_v4 port $webports
pass inet proto tcp to $emailserver_v4 port $email
pass log inet proto tcp from $emailserver_v4 to port smtp

pass inet6 proto tcp to $webserver_v6 port $webports
pass inet6 proto tcp to $emailserver_v6 port $email
pass log inet6 proto tcp from $emailserver_v6 to port smtp
```

- **NB:** No redirects are needed for IPv6

---
## Exercise 3 - checks

- Try connecting to the HTTP and SMTP port of your friends/neighbours:
 
- From Gateway:

```shell
    telnet -6 fd18:b5d:XX::80 80
    telnet -4 10.255.255.XX 80

```
 
- and

```shell
    telnet -6 fd18:b5d:XX::25 25
    telnet -4 10.255.255.XX 25

```

---

## Tips

- Decide your network topology
  - DMZ (?)
  - Multi-customer (?)
  - Multi-customer, Multi-DMZ(?)

- Segment off your subnets
  - IPv4 (Do you NAT)?
  - IPv6
  - Do you do NAT64?

- Per subnet (customer)
  - Which services do you expose?
  - Write the rules
  - pamper^H^H^H^H^Hproxying

---

# Traffic Shaping
  
  
### Section 4

---

## Traffic shaping

- OpenBSD has three separate shaping techniques:

- *priorities* (set prio), introduced in OpenBSD 5.0

- *queues*, introduced in OpenBSD 5.5, and

- *flows*, introduced in OpenBSD 6.2 (aka FQ-CoDel)
 
- **Remember:**

- Traffic shaping is about *dropping packets*

- But is only relevant when there's a *reason* to start dropping

- Then you get to pick which ones using the traffic shaping tools
 
- Works only one way, **outbound**

---
## Traffic shaping with priorities

- In OpenBSD, every packet has a priority
 
- Possible values are 0 (garbage) through 7 (want!)
 - Default for most traffic is 3.
 
- So if you want *all* ssh traffic to move ahead of other traffic you could do a
 
```shell
 pass proto tcp to port ssh set prio 6
```
 
- and assign specific, non-default ( != 3 ) values to others.

---
## Beating the FIFO with prio

- By default, packets are serviced on a first come, first served basis
 - or 'First in, First out' (FIFO).
 
- But: TCP wants ACKs for sent packets within a reasonable time
 - Otherwise the packet is considered lost, and retransmit will follow (transfer stalls).
 
- ACKs are tiny and have their TOS set to lowdelay, and this trick will cheat the FIFO:
 
```shell
 match out on $ext_if proto tcp from $ext_if set prio (3, 7)
 match in on $ext_if proto tcp to $ext_if set prio (3, 7)
```
 
as per the [man page](http://man.openbsd.org/pf.conf),
 
*If two priorities are given, TCP ACKs with no data payload and packets which have a TOS of lowdelay will be assigned to the second one. Packets with a higher priority number are processed first, and packets with the same priority are processed in the order in which they are received.*

- See [Prioritizing empty TCP ACKs with pf and ALTQ](http://www.benzedrine.ch/ackpri.html) by Daniel Hartmeier for the ALTQ way

---
## FQ-CoDel flows for fair bandwidth sharing

Introduced in [OpenBSD 6.2](https://www.openbsd.org/62.html), the FQ-CoDel algorithm (see [RFC8290](https://tools.ietf.org/html/rfc8290)) defines **flows** to set up fair sharing for a specified number of simultaneous connections

Enable for your configuration with something like

```shell
queue outq on bge0 bandwidth 18M max 18M flows 1024 qlimit 1024 \
               default
```

Estimate your approximate high number of simultaneously actively transmitting sessions, put that number in (up to 32767)

---
##  Shaping with HFSC queues - fixed sizes

- When priorities don't quite cut it, you can slice your bandwidth into queues.
 
- For static shaping, give bandwidth values in absolute values:
 
- Only leaf queues can be assigned traffic
 - make sure allocations sum up to parent queue allocation
 
- Unless quotas approach saturation, no actual shaping (dropping) will take place
 
---
## Shaping with HFSC queues - fixed sizes

```shell
  queue main on $ext_if bandwidth 20M
      queue defq parent main bandwidth 3600K default
      queue ftp parent main bandwidth 2000K
      queue udp parent main bandwidth 6000K
      queue web parent main bandwidth 4000K
      queue ssh parent main bandwidth 4000K
        queue ssh_interactive parent ssh bandwidth 800K
        queue ssh_bulk parent ssh bandwidth 3200K
  queue icmp parent main bandwidth 400K
```

---
##  Shaping with HFSC queues - fixed assignment

- You can either do queue assignment with match rules:
  
```shell
match log quick on $ext_if proto tcp to port ssh \
	queue (ssh_bulk, ssh_interactive)
match in quick on $ext_if proto tcp to port ftp queue ftp
match in quick on $ext_if proto tcp to port www queue http
match out on $ext_if proto udp queue udp
match out on $ext_if proto icmp queue icmp
```
- treat filtering (block, pass) in separate rules,

-  *or*

- append 'set queue' to individual pass rules

- Any traffic not explicitly assigned goes in the default queue
  
---
##  Shaping with HFSC queues - flexible allocations (min, max, burst)

You can build in flexibility:
  
```shell
queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K max 18M default
             queue qweb parent main bandwidth 9600K min 6000K max 18M
             queue qpri parent main bandwidth 700K min 100K max 1200K
             queue qdns parent main bandwidth 200K min 12K burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300
```
Note here the added *min* and *max* values: combinded queue bandwidth can exceed actual sum; this gets you upper and lower bound within physical limits.

*burst N for M* - allow bursts of that size and length

*qlimit* - size of the queue's holding buffer - larger values *may* delay (packets are kept longer before sending)

---
## Systat

- Available on all the BSDs
  - OpenBSD version has added functionalities

- Offers view into your system's traffic
  - Live queues, states, rules
  
- Check how your rules are behaving on your system in realtime

- More info in the man page [systat(1)](http://man.openbsd.org/systat)

---
## queue monitoring - systat queues

```shell
   1 users Load 2.56 2.27 2.28                        skapet.bsdly.net 20:55:50

QUEUE                BW SCH  PRI    PKTS   BYTES   DROP_P   DROP_B QLEN BOR SUS  P/S   B/S
rootq on bge0       20M                0       0        0        0    0            0     0
 main               20M                0       0        0        0    0            0     0
  qdef               9M          6416363   2338M      136    15371    0          462 30733
  qweb               9M           431590 144565K        0        0    0          0.6   480
  qpri               2M          2854556 181684K        5      390    0           79  5243
  qdns             100K           802874  68379K        0        0    0          0.6    52
 spamd               1K           596022  36021K  1177533 72871514  299            2   136
```  
or

---
## queue monitoring - pfctl

```shell
$ doas pfctl -vsq
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue rootq on bge0 bandwidth 20M qlimit 50
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue main parent rootq bandwidth 20M, min 1M, max 20M qlimit 100
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
queue qdef parent main bandwidth 9M, min 8M, max 18M default qlimit 50
  [ pkts:    6517150  bytes: 2458545319  dropped pkts:    136 bytes:  15371 ]
  [ qlength:   0/ 50 ]
queue qweb parent main bandwidth 9M, min 8M, max 18M qlimit 50
  [ pkts:     431741  bytes:  148072219  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue qpri parent main bandwidth 2M, min 700K, max 2M burst 4M for 3000ms qlimit 50
  [ pkts:    2855418  bytes:  186101241  dropped pkts:      5 bytes:    390 ]
  [ qlength:   0/ 50 ]
queue qdns parent main bandwidth 100K, min 12K burst 600K for 3000ms qlimit 50
  [ pkts:     803548  bytes:   70079760  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue spamd parent rootq bandwidth 1K, max 1K qlimit 300
  [ pkts:     596424  bytes:   36910940  dropped pkts: 1178456 bytes: 72928604 ]
  [ qlength: 300/300 ]
```

add another v ([pfctl](http://man.openbsd.org/pfctl) -vvsq) for continuously updating display
 
---

# Questions ?
  
???

Let's ask if there are any questions before continuing.  Make sure we have everyone onboard.

---

# Exercise 4

### Queueing

---

##  Exercise 4 - Goals

- With the configs from exercise 3, now add:
 
- A set of queues, and
 
- Statements to add rules to the queues

---

## Exercise 4 - on Gateway

- Configure the queues

### /etc/pf.conf

```shell
queue rootq on $ext_if bandwidth 20M
    queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100
        queue default parent main bandwidth 9600K min 6000K max 18M default
        queue http parent main bandwidth 9600K min 6000K max 18M
        queue smtp parent main bandwidth 9600K min 6000K max 18M
    queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300
```

---
## Exercise 4 - on Gateway

- and then apply them to the match statements

### /etc/pf.conf

```shell
match in on egress inet proto tcp to egress port $webports rdr-to $webserver_v4 \
      queue http
match in on egress inet proto tcp to egress port $email rdr-to $emailserver_v4 \
      queue smtp

pass inet6 proto tcp to $webserver_v6 port $webports set queue http
pass inet6 proto tcp to $emailserver_v6 port $email set queue smtp
pass log inet6 proto tcp from $emailserver_v6 to port smtp set queue smtp

```

---

## Exercise 4 - Check

- Check the queues have been effectively created

```shell
 # systat queues

```
 
- or, alternatively

```shell
 # pfctl -vsq

```
---

# Tips
  
  
### Section 5

---
# Choosing your ISP, a quick guide

- Are they national or regional IX members?
 
- Do they have geographical redundancy ?
 - or do you need to arrange that for yourself ?
 
- Do they actually understand your questions about peering, routing, multiple paths?
 - (avoid consumer oriented SOHO-only shops)
 
- Do they _suck_?

---
## Getting transit

- Find well peered transit providers
 - Can improve quality and shorten AS paths
 - No capacity problems
 
- Find your top traffic destinations:
 - Can improve quality
 - Peer with them or find closer upstream
 - Traffic profile from flow collectors can be useful

---

## Common mistakes

- No diversity
 - All reached over same cable
 - All connect to the same transit
 - All have poor onward transit and peering arrangements 
 
- Signing up with too many transit providers
 - Lots of small circuits
 - These cost more per Mbps than larger ones

---
##   Basic OpenBGPd configuration, operation and interaction with PF
  
  - **B**order **G**ateway **P**rotocol
    - Manage and exchange route information with BGP peers

- Once you have the ASn registered, do the basic config. 

 - In your *pf.conf*:
 - enable BGP to pass between your routers and your peers' -- **TCP and UDP 179**

- **Neat trick**: Define tables in your [pf.conf](http://man.openbsd.org/pf.conf)
    - bgpd maintains them via **pftable** attributes on [bgpd.conf](http://man.openbsd.org/bgpd.conf) objects

---
## Use cases for OSPF, BGP or ECMP

- **OSPF:** **O**pen **S**hortest **P**ath **F**irst 
  - is a IGP **I**interior **G**ateway **P**rotocol
  - Each router maintains link state information for links and networks within your AS
  - Calculates routing cost
  - Use [ospf6d](http://man.openbsd.org/ospf6d) for IPv6
  - Use [ospfd](http://man.openbsd.org/ospfd) for IPv4
  - Need to *pass proto ospf* between routers.

- **BGP:** announces and receives routes
  - can be both an IGP or EGP **E**xterior **G**ateway **P**rotocol
  - highly scalable (Internet scale)
  - can be used for signaling and sending additional information with route announcements
  - Use [bgpd](http://man.openbsd.org/bgpd)
  - need to *pass proto tcp port 179* between routers

---
## Use cases for OSPF, BGP or ECMP (cont)

- **ECMP:** **E**qual **C**ost **M**ulti-**P**ath
  - target reachable via more than one route
  - load distribution or redundancy over multiple links
  - **Tip** Use [ifstated](http://man.openbsd.org/ifstated) to handle link downtime.

---
## BCP38, MANRS and Internet peering

"[**BCP38**](https://tools.ietf.org/html/bcp38)" -- Discussed also in another effort
  
**M**utually **A**greed **N**orms for **R**outing **S**ecurity (MANRS)

- Define four concrete actions network operators should implement
- Coordination
   - Keep your contacts updated
- Validation
   - Route objects, RPKI, BGPSec
- Anti-spoofing  
 - uRPF
 - Filtering on external Interfaces facing external suppliers  
    - Drop inbound Traffic with a src IP claiming to be from your networks / private networks.
    - Drop outbound Traffic with a src IP address that is not  in your Public IP network range.
- Build a visible community of security-minded operators
- Valuable resource: [The Routing Manifesto](https://www.routingmanifesto.org/)

---
##    Introducing VXLAN in your network

[vxlan](http://man.openbsd.org/vxlan) - the **V**irtual e**X**tensible **L**ocal **A**rea **N**etwork tunnel interface
 
- Pushes layer 2 network (Ethernet frames) over layer 3 (IP) tunnels
 - 24-bit *vnetid* (vs max 4k VLANs)
 
- Has *no* built in security
- Intended for '*trusted*' (Datacenter, inter-hypervisor) environments
 - Otherwise, consider transport over IPSEC.
 
- Default transport over **UDP 4789** (aka **vxlan**)
 - make sure that traffic passes between endpoints

---
#    Introducing VXLAN in your network

```shell
 # ifconfig vxlan0 tunnel 192.168.100.101 192.168.200.201 vnetid 17
 # ifconfig vxlan0 10.11.12.100/24

```
 
```shell
 # ifconfig vxlan0 tunnel 192.168.200.201 192.168.100.101 vnetid 17
 # ifconfig vxlan0 10.11.12.101/24

```
 
```shell
table <vxendpoints> { 192.168.200.201 192.168.200.204 }
pass from <vxendpoints> to port vxlan
```
 
Buy [Reyk](http://www.bsdcan.org/2017/schedule/speakers/227.en.html) a beer.

---
## Readable and maintainable toolsets

- **Macros**
  - descriptive names, keep uniform
- **Tables** 
  - descriptive names
  - consider daemon/scripting interface
- **Interface groups**
  - you know egress already
  - make your own and filter on them
- **Anchors**
  - group rules by common criteria
  - tagging 
  - interface or group
- Service names vs port numbers

- **Comments** - yes, you **will** forget why this was a good idea

---
## Useful 3rd party packages (ports) for OpenBSD

OpenBSD base operating system can be supplimented by the following packages and features:

- pftop - a curses-based utility for real-time display of
active states and rules for pf. It is a cross between top
and pfctl -sr and pfctl -ss.
  - pftop can be installed with the following command

<code>pkg_add pftop</code>
 
 - nsh **n**etwork **sh**ell 
 - nsh can be installed with the following command
 
 <code>pkg_add nsh</code>

---
## Now let's add wireless

- Wireless used to be hard, (WPA in particular), now it's 'just another interface'
 
- 802.11* support in OpenBSD has a,b,g,n, ac only in some drivers ([bwfm(4)](https://man.openbsd.org/bwfm), [iwx(4)](https://man.openbsd.org/iwx))
 
- Not all drivers support hostap
 - check man pages before buying kit for access point use
 
- Optionally setup with commercial APs for radio part
 - do DHCP, filtering, authentication and so forth from OpenBSD

---
class: center, middle

# Questions ?
  
???

Let's ask if there are any questions before continuing.  Make sure we have everyone onboard.

---

# Troubleshooting
  
  
### Section 6

#### "It's all your fault. Until you track down and fix the root cause."
---
## Troubleshooting 101: ICMP(v6)

- ICMP: **I**nternet **C**ontrol **M**essage **P**rotocol

- The *ping of death* scare is almost over, let's enable [ping](http://man.openbsd.org/ping):
 
```shell 
 icmp_types = "{ echoreq, unreach }"
 
 pass inet proto icmp all icmp-type $icmp_types keep state
 
 pass inet proto icmp from $localnet icmp-type $icmp_types
 pass inet proto icmp to $ext_if icmp-type $icmp_types
 pass inet6 proto icmp6 from $localnet icmp6-type $icmp6_types
 pass inet6 proto icmp6 to $ext_if icmp6-type $icmp6_types
```
 
- **echoreq**: lets [ping](http://man.openbsd.org/ping) do its thing

- **unreach**: lets you do _path MTU discovery_ (PMTUD)

---
## Troubleshooting 101: ICMP not all messages created equal!

- Some ICMP messages are frivolous:
- Type 4 — Source Quench (Deprecated)
- Type 5 — Redirect (if you need this you are doing it wrong) 
- Type 6 — Alternate Host Address (Deprecated)
- Type 13 — Timestamp
- Type 14 — Timestamp Reply
- Type 15 — Information Request (Deprecated)
- Type 16 — Information Reply (Deprecated)
- Type 17 — Address Mask Request (Deprecated)
- Type 18 — Address Mask Reply (Deprecated)
---
## Troubleshooting 101: basic diagnostics built in in OpenBSD base
Basic IP connectivity 
```shell 
ping <host>
```
```shell 
traceroute <host>
```
```shell 
telnet <host> <tcp port>
```
```shell 
tcpdump -i <interfacename> 
```
 - netcat command - nc 
```shell 
 nc -z <host> <startport>-<endport>
```
---
## Troubleshooting 101: not all pings were created equal! 
ping packets can be manipulated in many ways
- set a specific source address (useful for routers)
```shell
ping [-I <src>address> ] <destaddress>
```
-set ping packet size (MTU path Discovery Issues) 
```shell
ping [-s <packet-size>] <destaddress>
```
- set dont fragment bit (MTU / Fragmentation issues) 
```shell
ping [ -D ] <destaddress>
```
-set ping packet size (MTU / fragmentation issues
```shell
ping [-s <packet-size>] <destaddress>
```
---
## Troubleshooting 101: basic diagnostics built in in OpenBSD base
DNS functionality
basic commands

```shell 
ping <hostdnsnae>
```

```shell 
telnet <hostdnsnae> <tcp port>
```
- in depth dns testing 
```shell 
dig <hostdnsname>
```
```shell 
nslookup <hostdnsname>
```
---
## Troubleshooting 101: diagnostic tools available in ports 
IP connectivity
- mtr (not the gtk version) 
```shell 
mtr <host>
```
- tcp traceroute
```shell 
tcptraceroute <host> 
```
---
## Troubleshooting 101: Statistics

- Statistics can be had with **[pfctl](http://man.openbsd.org/pfctl) -s info**

For statistics (bytes/packets passed per rule) attach _labels_ per rule

```shell 
pass log proto { tcp, udp } to $emailserver port smtp label "mail-in"
pass log proto { tcp, udp } from $emailserver to port smtp label "mail-out"
```

```shell
$ doas pfctl -vs rules
pass inet proto tcp from any to 192.0.2.225 port = smtp flags S/SA keep state label "mail-in"
[ Evaluations: 1664158 Packets: 1601986 Bytes: 763762591 States: 0 ]
[ Inserted: uid 0 pid 24490 ]
pass inet proto tcp from 192.0.2.225 to any port = smtp flags S/SA keep state label "mail-out"
[ Evaluations: 2814933 Packets: 2711211 Bytes: 492510664 States: 0 ]
[ Inserted: uid 0 pid 24490 ] 
```

---
## Troubleshooting 101: Statistics

- If you need to pass the data to a script
 - Or a database
 - A graphing engine

```shell
$ doas pfctl -zvsl
mail-in 1664158 1601986 763762591 887895 682427415 714091 81335176
mail-out 2814933 2711211 492510664 1407278 239776267 1303933 252734397
```

---
## Troubleshooting 101: log to pflog

Rules with the **log** keyword log packet data to the [pflog](http://man.openbsd.org/pflog) device(s)

```shell
# log blocked packets
block log(all)

# logs initial packet of matching connections:	
pass log proto tcp to port ssh
   
# logs all matching packets:
pass log(all) proto tcp to port ssh log(all)

# logs matches on this and all succeeding rules
pass log(matches) proto tcp to port ssh

# logs all packets matches on this and all succeeding rules
pass log(all, matches) proto tcp to port ssh  
```

```shell
match log(all, matches) # log *everything*
```

---
## Troubleshooting 101: tcpdump, read from pflog

- [tcpdump](https://man.openbsd.org/tcpdump) is your friend

- Let it loose on the pflog device:

```shell
$ doas tcpdump -n -e -ttt -i pflog0 
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
May 29 21:06:27.165561 rule def/(match) pass in on bge1: 192.168.103.126.15526 >
213.187.179.198.22: . ack 2951513182 win 16332 (DF) [tos 0x10]
May 29 21:06:27.166934 rule 16/(match) pass in on bge0: 158.36.191.135.22 >
213.187.179.198.59516: . ack 1734404306 win 64800 [tos 0x8]
May 29 21:06:27.166939 rule 2/(match) match in on bge0: 158.36.191.135.22 >
213.187.179.198.59516: . ack 1 win 64800 [tos 0x8]
May 29 21:06:27.168340 rule def/(match) pass out on bge1: 213.187.179.198.22 >
192.168.103.126.15526: P 69:153(84) ack 0 win 17520 [tos 0x10]
May 29 21:06:27.169150 rule def/(match) pass out on bge1: 213.187.179.198.22 >
192.168.103.126.15526: P 153:333(180) ack 0 win 17520 [tos 0x10]
May 29 21:06:27.169265 rule def/(match) pass out on bge1: 213.187.179.198.22 >
```

- **NB** rule number, matches your *loaded* rule set

---
## Troubleshooting 101: Hitting and avoiding limits

- On busy systems, you may need to raise limits from default values

- Check with:

```shell
$ doas pfctl -s info
```
- versus the output of **pfctl -s memory** and **pfctl -s timeouts**

- You may need to bump up from defaults:

```shell
# increase state limit from 10'000 states on busy systems
set limit states 100000
# increase no of source nodes 
set limit src-nodes 100000
```

---
## Troubleshooting 101: quick flow analysis using iftop (package)

- Sometimes we need to check what traffic is flowing on a given interface

- iftop ncurses package can be installed and allows visibility on top traffic flows on an interface

- iftop allows that quick diagnostics check without needing a full flow anlaysis infrastructure

- iftop ncurses package can be installed on any OpenBSD system using the following command

```shell 
pkg_add iftop
```

---
## Troubleshooting 101: using iftop (package) example

- Checking flow on the first intel 1Gbs interface on a router
```shell
iftop -i em0
```
- command output
```shell
 1.91Mb 3.81Mb 5.72Mb 7.63Mb 9.54Mb
└───────────────┴───────────────┴───────────────┴───────────────┴───────────────
10.0.2.15 => di-in-f190.1e100.net 23.5Kb 11.8Kb 2.95Kb
 <= 3.22Mb 805Kb 201Kb
10.0.2.15 => dj-in-f106.1e100.net 0b 18.1Kb 6.53Kb
 <= 0b 708Kb 218Kb
10.0.2.15 => dj-in-f94.1e100.net 0b 4.41Kb 1.10Kb
 <= 0b 57.5Kb 14.4Kb
10.0.2.15 => dj-in-f100.1e100.net 0b 4.14Kb 1.03Kb
 <= 0b 43.3Kb 10.8Kb
10.0.2.15 => dh-in-f95.1e100.net 5.69Kb 4.36Kb 1.60Kb
 <= 133Kb 29.5Kb 8.06Kb
10.0.2.15 => dj-in-f119.1e100.net 0b 3.26Kb 834b
 <= 0b 19.6Kb 4.91Kb
10.0.2.15 => ig-in-f94.1e100.net 0b 4.66Kb 1.16Kb
 <= 0b 12.8Kb 3.21Kb
10.0.2.15 => dj-in-f148.1e100.net 12.5Kb 2.49Kb 638b
 <= 49.9Kb 9.99Kb 2.50Kb
10.0.2.15 => dg-in-f102.1e100.net 0b 6.57Kb 1.64Kb
 <= 0b 5.51Kb 1.38Kb
────────────────────────────────────────────────────────────────────────────────
TX: cum: 832KB peak: 89.6Kb rates: 51.7Kb 62.4Kb 20.1Kb
RX: 20.6MB 3.43Mb 3.43Mb 1.66Mb 472Kb
TOTAL: 21.4MB 3.48Mb 3.48Mb 1.72Mb 492Kb
```
---
## Troubleshooting 101: netflow aka pflow (IPFIX)

- Records TCP/IP *flow* metadata
  - srcIP
  - dstIP 
  - (srcPort, dstPort)
  - startTime
  - endTime 
  - Packets
  - Bytes

- OpenBSD has the [pflow(4)](http://man.openbsd.org/pflow) virtual network interface
  - which generates the datagrams from the state table

- Useful for network monitoring, DDoS protection, etc.

---
## Troubleshooting 101: netflow setup

- Set up a *sensor*:

```shell
$ cat /etc/hostname.pflow0
flowsrc 192.168.103.1 flowdst 192.168.103.252:9995
pflowproto 10
```

- Then configure your *collector* at the **flowdst** IP address for analysis and network overlordship.

- Lots of collector options available in ports: nfsen, flow-tools, pmacct, FastNetMon and others.

- More info:
  - Michael W. Lucas: [Network Flow Analysis](https://www.nostarch.com/networkflow)
  - and Peter N. M. Hansteen: [Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen](http://bsdly.blogspot.ca/2014/02/yes-you-too-can-be-evil-network.html).

---
## Flow Analyser example Fastnetmon 
- Example of a typcial flow anlayser software fastnetmon:
  - User can view FastNetMon statistics via the CLI client fastnetmon_client
```shell
# fastnetmon_client
FastNetMon 1.1.7 master git- Try Advanced edition: https://fastnetmon.com
IPs ordered by: packets
Incoming traffic       1505664 pps  15397 mbps     85 flows
37.203.[redacted]        59184 pps    485 mbps      0 flows
37.203.[redacted]        45040 pps    504 mbps      0 flows
37.203.[redacted]        26924 pps    270 mbps      0 flows
185.55.[redacted]        24211 pps    240 mbps      0 flows
5.134.[redacted]         23872 pps    290 mbps      0 flows
45.11.[redacted]         23634 pps    250 mbps      0 flows
185.55.[redacted]        22451 pps    255 mbps      0 flows
45.11.[redacted]         20943 pps    254 mbps      0 flows
185.55.[redacted]        20298 pps    246 mbps      0 flows
5.134.[redacted]         20188 pps    236 mbps      0 flows
```
- With FastNetMon one can implement mitigations based on tresholds
  - Packets per second pps
  - Bandwidth per second Mbps 
---

# Exercise 5

### NAT64

---

##  NAT64

- Single-stack clients will only have IPv6

- Translator box will strip all headers and replace them with IPv4
 
- Requires some DNS “magic”
 - Capture responses and replace A with AAAA
 - Response is crafted based on target IPv4 address

- Usually implies address sharing on IPv4

---
background-image: url(images/nat64.png)

---

## 464-XLAT

- Extension to NAT64 to access IPv4-only applications (like Skype or Whatsapp)

- Handset pretends there is an IPv4 address (CLAT) and sends IPv4 packets in UDP over IPv6

---
background-image: url(images/464-xlat.png)

---

## Exercise 5 - Goals

- Define the translating prefix
  - 64:ff9b::/96 is reserved by IETF

- Add NAT64 configuration to PF

- Remove IPv4 from the internal network

- Configure DNS64 on unbound

---

## Exercise 5

- _pf.conf_

```shell
pass in quick on $int_if inet6 from any to 64:ff9b::/96 \
   af-to inet from (egress:0) keep state
```

- _unbound.conf_

```shell
module-config: "dns64 validator iterator"

dns64-prefix: 64:FF9B::/96

```

and it's done!

---

# Questions ?

### Last chance...

### or questions@pftutorial.net

???

Let's ask if there are any questions before continuing.  Make sure we have everyone onboard.

---

## Web accessible resources
### OpenBSD website and documentation

[http://www.openbsd.org/](http://www.openbsd.org/)  The official OpenBSD website – to donate: [http://www.openbsd.org/donations.html](http://www.openbsd.org/donations.html) and please do donate, corporates may prefer [https://www.openbsdfoundation.org/](https://www.openbsdfoundation.org/)  - a Canadian non-profit

[The PF User Guide on the OpenBSD web site](http://www.openbsd.org/faq/index.html)

[OpenBSD online man pages](http://man.openbsd.org/)

Note: You can convert the man page of pf.conf to PDF for reading in your favourite reader with the command:

---
## Resources

### Books / e-Books 
Michael W Lucas: [Absolute OpenBSD, 2nd ed.](https://www.nostarch.com/openbsd2e)

Peter N. M. Hansteen: [The Book of PF, 3rd ed.](https://www.nostarch.com/pf3)

Elizabeth D. Zwicky, Simon Cooper, D. Brent Chapman [Building Internet Firewalls, 2nd ed.](https://www.oreilly.com/library/view/building-internet-firewalls/)

### Blogs
[http://undeadly.org/](http://undeadly.org/)  - The OpenBSD Journal news site

[http://bsdly.blogspot.com/](http://bsdly.blogspot.com/)  - Peter's rants^H^H^H^H^Hblog posts

[http://www.tedunangst.com/flak/](http://www.tedunangst.com/flak/)  tedu@ on developments

---
background-image: url(images/end.png)
 
-

???

Notes for this slide    
  
---
background-image: url(images/end2.png)

- 
  
???

Notes for this other slide

---