GKE上で動くPodでeBPFを使ってみる

GKEのバージョン

1.16.13-gke.1

Node

Container-Optimized OS from Google

コンテナのOS

OSは下記を使うのでちょっと特殊

https://console.cloud.google.com/gcr/images/gcp-runtimes/GLOBAL/ubuntu_18_0_4?gcrImageListsize=30

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

# uname -r
4.19.112+

コンテナのbpfの状態

zgrep -i bpf /proc/config.gz
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_BPFILTER is not set
CONFIG_NET_CLS_BPF=m
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_JIT=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
CONFIG_TEST_BPF=m

準備

Podの設定

privileged modeを有効にしておく

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged

securityContext:
  privileged: true

Podの中での作業

bcc/INSTALL.md at master · iovisor/bcc · GitHub を参考にしながらやってみる

wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.112/linux-headers-4.19.112-0419112_4.19.112-0419112.202003200739_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.112/linux-headers-4.19.112-0419112-generic_4.19.112-0419112.202003200739_amd64.deb
dpkg -i *.deb
mkdir -p /lib/modules/4.19.112+/
# 確認コマンド
ls /lib/modules/4.19.112-0419112-generic/build
ln -s /lib/modules/4.19.112-0419112-generic/build /lib/modules/4.19.112+/build

#  https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---binary 参照してiovisor packages (Upstream Stable and Signed Packages) をいれる
apt-get install gnupg
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/iovisor.list
apt-get update
apt-get install bcc-tools

結果その1 (privileged modeが無効の場合)

# /usr/share/bcc/tools/execsnoop
could not open bpf map: events, error: Operation not permitted
Traceback (most recent call last):
  File "/usr/share/bcc/tools/execsnoop", line 165, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 325, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text

権限不足になる

結果その2 (privileged modeが有効の場合)

# /usr/share/bcc/tools/execsnoop
PCOMM            PID    PPID   RET ARGS
gke              1684128 ?        0 /home/kubernetes/bin/gke
portmap          1684133 ?        0 /home/kubernetes/bin/portmap
bandwidth        1684137 ?        0 /home/kubernetes/bin/bandwidth
timeout          1684141 ?        0 /usr/bin/timeout 60 docker ps

おおおおお

うまくいった