びぼうろぐ

ネ申エクセルからの卒業

【Gitlab】Office365にメールが届かない

公式リファレンスの設定例を参考にOffice365へSMTPを送信するように設定したつもりでしたが、メールが届きませんでした。

https://docs.gitlab.com/omnibus/settings/smtp.html#office365

その時に調べた内容をまとめます。

環境情報

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

$ uname -r
4.18.0-305.17.1.el8_4.x86_64

$ sudo gitlab-rake gitlab:env:info
System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.32.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.2.3-ee
Revision:       b5eea856eca
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            http://<mygitlab>
HTTP Clone URL: http://<mygitlab>some-group/some-project.git
SSH Clone URL:  git@<mygitlab>:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.19.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

再現例

①. コンフィグファイルを編集する。

$ sudo /etc/gitlab/gitlab.rb
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.office365.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "username@yourdomain.com"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "yourdomain.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['gitlab_email_from'] = 'username@yourdomain.com'

②. 設定を反映させる。

$ sudo gitlab-ctl reconfigure
$ sudo gitlab-ctl restart

③. ログイン画面の「Forgot your password?」から登録済みのメールアドレスを入力し、パスワードリセットの要求を出す。

④. ③で入力したメールの受信箱を確認する。

解決方法

/etc/gitlab/gitlab.rbの設定であるgitlab_rails['smtp_tls'] = trueコメントアウト(もしくは明示的にfalseを指定)し、sudo gitlab-ctl reconfigure & restertする。
たしかに公式にあるO365用の設定例にはgitlab_rails['smtp_tls']自体載ってないんですね...

エラーの原因

ログに吐かれているエラー内容で探すと以下のページがヒットしました。

https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4253

ということで以下の2つが原因みたいです。

  • OpenSSLはデフォルトで「SSLv2/v3」を使おうとする。
  • Office365のSMTPサーバは「SSLv2/v3」の接続を受け付けない。

ログの出力内容

$ sudo less /var/log/gitlab/gitlab-rails/production.log
...
[ActiveJob] [ActionMailer::MailDeliveryJob] [8bb40c1a-0bd2-4e24-82b9-322822565db2] Performing ActionMailer::MailDeliveryJob (Job ID: 8bb40c1a-0bd2-4e24-82b9-322822565db2) from Sidekiq(mailers) enqueued at 2021-09-13T08:40:51Z with arguments: "DeviseMailer", "reset_password_instructions", "deliver_now", {:args=>[#<GlobalID:0x00007fe4ca987540 @uri=#<URI::GID gid://gitlab/User/2>>, "3BUa5ftTstC4zXhVfZxu", {}]}
[ActiveJob] [ActionMailer::MailDeliveryJob] [8bb40c1a-0bd2-4e24-82b9-322822565db2]   Rendered layout layouts/mailer/devise.html.haml (Duration: 11.8ms | Allocations: 4849)
[ActiveJob] [ActionMailer::MailDeliveryJob] [8bb40c1a-0bd2-4e24-82b9-322822565db2]   Rendered layout ee/app/views/layouts/mailer/devise.text.erb (Duration: 1.3ms | Allocations: 494)
[ActiveJob] [ActionMailer::MailDeliveryJob] [8bb40c1a-0bd2-4e24-82b9-322822565db2] Delivered mail 613f0f4aeef69_2ff52b1d858174@gl.mail (40.0ms)
[ActiveJob] [ActionMailer::MailDeliveryJob] [8bb40c1a-0bd2-4e24-82b9-322822565db2] Error performing ActionMailer::MailDeliveryJob (Job ID: 8bb40c1a-0bd2-4e24-82b9-322822565db2) from Sidekiq(mailers) in 64.98ms: OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: wrong version number):
...

(おまけ)実際の通信を見てみる

パケットキャプチャしつつどんな通信が流れていたかを確認します。
確認にはtshark(WiresharkCLI)を利用します。

tsharkのインストール

$ sudo dnf install -y wireshark
$ tshark -v
TShark (Wireshark) 2.6.2 (v2.6.2)
...

gitlab_rails['smtp_tls'] = true の場合

SSHRHELにアクセスしているので、22ポート以外の通信をキャプチャします。

$ sudo tshark -i eth0 -f "not port 22"
...
    9 0.667463513 192.168.0.1  → 192.168.0.6  DNS 248 Standard query response 0x9532 A smtp.office365.com CNAME outlook.office365.com CNAME outlook.ha.office365.com CNAME outlook.ms-acdc.office.com CNAME HND-efz.ms-acdc.office.com A 40.101.144.98 A 52.98.74.178 A 52.98.41.130 A 52.98.83.178
   10 0.671961837 192.168.0.1 → 192.168.0.6  DNS 296 Standard query response 0x33cd AAAA smtp.office365.com CNAME outlook.office365.com CNAME outlook.ha.office365.com CNAME outlook.ms-acdc.office.com CNAME HND-efz.ms-acdc.office.com AAAA 2603:1046:c09:1804::2 AAAA 2603:1046:c09:101e::2 AAAA 2603:1046:c09:1803::2 AAAA 2603:1046:404::2
   11 0.672228238  192.168.0.6 → 40.101.144.98 TCP 74 55588 → 587 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1374903688 TSecr=0 WS=128
   12 0.677717367 40.101.144.98 → 192.168.0.6  TCP 66 587 → 55588 [SYN, ACK] Seq=0 Ack=1 Win=8190 Len=0 MSS=1330 WS=16 SACK_PERM=1
   13 0.677762368  192.168.0.6 → 40.101.144.98 TCP 54 55588 → 587 [ACK] Seq=1 Ack=1 Win=29312 Len=0
   14 0.683858100 40.101.144.98 → 192.168.0.6  SMTP 165 S: 220 TYXPR01CA0046.outlook.office365.com Microsoft ESMTP MAIL Service ready at Mon, 13 Sep 2021 14:16:49 +0000
   15 0.683960200  192.168.0.6 → 40.101.144.98 TCP 54 55588 → 587 [ACK] Seq=1 Ack=112 Win=29312 Len=0
   16 0.684458903  192.168.0.6 → 40.101.144.98 SMTP 571 # 文字化けして見えない;ω;
   17 0.684631204  192.168.0.6 → 40.101.144.98 TCP 54 55588 → 587 [RST, ACK] Seq=518 Ack=112 Win=29312 Len=0
   18 0.775265781  192.168.0.6 → 192.168.0.254 TCP 7354 HTTP/1.1 200 OK  [TCP segment of a reassembled PDU]
   19 0.775273981  192.168.0.6 → 192.168.0.254 HTTP 3098 HTTP/1.1 200 OK  (text/html)
   20 0.777086990 192.168.0.254 → 192.168.0.6  TCP 54 61149 → 80 [ACK] Seq=2079 Ack=11171 Win=8212 Len=0

SMTPSTARTTLSしようとしているところでエラーになっている(?)ことが確認できました。
ClientHello時にSSLv2/v3で要求を出してOffice365側からNG出されたっぽいですね。

gitlab_rails['smtp_tls'] = false の場合

$ sudo tshark -i eth0 -f "not port 22"

...
   21 19.437202621 192.168.0.1 → 192.168.0.6  DNS 248 Standard query response 0x8cbc A smtp.office365.com CNAME outlook.office365.com CNAME outlook.ha.office365.com CNAME outlook.ms-acdc.office.com CNAME HND-efz.ms-acdc.office.com A 40.101.147.114 A 52.98.83.2 A 52.98.41.162 A 40.100.52.2
   22 19.437636223  192.168.0.6 → 40.101.147.114 TCP 74 37430 → 587 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1712271993 TSecr=0 WS=128
   23 19.450790794 40.101.147.114 → 192.168.0.6  TCP 66 587 → 37430 [SYN, ACK] Seq=0 Ack=1 Win=8190 Len=0 MSS=1330 WS=16 SACK_PERM=1
   24 19.450834594  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=1 Ack=1 Win=29312 Len=0
   25 19.457396830 40.101.147.114 → 192.168.0.6  SMTP 165 S: 220 TYAPR01CA0212.outlook.office365.com Microsoft ESMTP MAIL Service ready at Mon, 13 Sep 2021 13:25:24 +0000
   26 19.457463230  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=1 Ack=112 Win=29312 Len=0
   27 19.457543930  192.168.0.6 → 40.101.147.114 SMTP 82 C: EHLO jrsdc.onmicrosoft.com
   28 19.463651763 40.101.147.114 → 192.168.0.6  SMTP 259 S: 250-TYAPR01CA0212.outlook.office365.com Hello [218.221.56.19] | 250-SIZE 157286400 | 250-PIPELINING | 250-DSN | 250-ENHANCEDSTATUSCODES | 250-STARTTLS | 250-8BITMIME | 250-BINARYMIME | 250-CHUNKING | 250 SMTPUTF8
   29 19.463877264  192.168.0.6 → 40.101.147.114 SMTP 64 C: STARTTLS
   30 19.501085964 40.101.147.114 → 192.168.0.6  SMTP 83 S: 220 2.0.0 SMTP server ready
   31 19.504957985 40.101.147.114 → 192.168.0.6  TCP 83 [TCP Retransmission] 587 → 37430 [PSH, ACK] Seq=317 Ack=39 Win=1048560 Len=29
   32 19.505018085  192.168.0.6 → 40.101.147.114 TCP 66 37430 → 587 [ACK] Seq=39 Ack=346 Win=30336 Len=0 SLE=317 SRE=346
   33 19.506490593  192.168.0.6 → 40.101.147.114 TLSv1 571 Client Hello
   34 19.513486331 40.101.147.114 → 192.168.0.6  TCP 1384 [TCP segment of a reassembled PDU]
   35 19.548130617 40.101.147.114 → 192.168.0.6  TCP 1384 [TCP Retransmission] 587 → 37430 [ACK] Seq=346 Ack=556 Win=1048560 Len=1330
   36 19.548189517  192.168.0.6 → 40.101.147.114 TCP 66 37430 → 587 [ACK] Seq=556 Ack=1676 Win=33280 Len=0 SLE=346 SRE=1676
   37 19.554056049 40.101.147.114 → 192.168.0.6  TCP 1384 587 → 37430 [ACK] Seq=1676 Ack=556 Win=1048560 Len=1330 [TCP segment of a reassembled PDU]
   38 19.554113549  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=556 Ack=3006 Win=36224 Len=0
   39 19.591069048 40.101.147.114 → 192.168.0.6  TLSv1.2 1250 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
   40 19.591100248  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=556 Ack=4202 Win=39040 Len=0
   41 19.591247249 40.101.147.114 → 192.168.0.6  TCP 1384 [TCP Out-Of-Order] 587 → 37430 [ACK] Seq=1676 Ack=556 Win=1048560 Len=1330[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
   42 19.591259149  192.168.0.6 → 40.101.147.114 TCP 66 [TCP Dup ACK 40#1] 37430 → 587 [ACK] Seq=556 Ack=4202 Win=39040 Len=0 SLE=1676 SRE=3006
   43 19.593598261  192.168.0.6 → 40.101.147.114 TLSv1.2 224 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
   44 19.595509072 40.101.147.114 → 192.168.0.6  TCP 1250 [TCP Spurious Retransmission] 587 → 37430 [PSH, ACK] Seq=3006 Ack=556 Win=1048560 Len=1196[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
   45 19.595534372  192.168.0.6 → 40.101.147.114 TCP 66 [TCP Dup ACK 40#2] 37430 → 587 [ACK] Seq=726 Ack=4202 Win=39040 Len=0 SLE=3006 SRE=4202
   46 19.601969806 40.101.147.114 → 192.168.0.6  TLSv1.2 105 Change Cipher Spec, Encrypted Handshake Message
   47 19.602002706  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=726 Ack=4253 Win=39040 Len=0
   48 19.602355808  192.168.0.6 → 40.101.147.114 TLSv1.2 111 Application Data
   49 19.609491847 40.101.147.114 → 192.168.0.6  TLSv1.2 298 Application Data
   50 19.609758348  192.168.0.6 → 40.101.147.114 TLSv1.2 95 Application Data
...
   73 20.279348345 40.101.147.114 → 192.168.0.6  TCP 54 587 → 37430 [ACK] Seq=4835 Ack=3867 Win=1048560 Len=0
   74 20.279445946  192.168.0.6 → 40.101.147.114 TLSv1.2 868 Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data, Application Data
   75 20.279462546 40.101.147.114 → 192.168.0.6  TCP 54 587 → 37430 [ACK] Seq=4835 Ack=7857 Win=1048560 Len=0
   76 20.288288993 40.101.147.114 → 192.168.0.6  TCP 54 587 → 37430 [ACK] Seq=4835 Ack=10001 Win=1048560 Len=0
   77 20.833251421 40.101.147.114 → 192.168.0.6  TLSv1.2 187 Application Data
   78 20.833484622  192.168.0.6 → 40.101.147.114 TLSv1.2 89 Application Data
   79 20.848379002 40.101.147.114 → 192.168.0.6  TLSv1.2 131 Application Data
   80 20.848640304  192.168.0.6 → 40.101.147.114 TLSv1.2 85 Encrypted Alert
   81 20.848653804  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [FIN, ACK] Seq=10067 Ack=5045 Win=44416 Len=0
   82 20.855102038 40.101.147.114 → 192.168.0.6  TCP 54 587 → 37430 [ACK] Seq=5045 Ack=10068 Win=1048560 Len=0
   83 20.885093099 192.168.0.254 → 192.168.0.6  TCP 54 54793 → 80 [FIN, ACK] Seq=1 Ack=1 Win=2102272 Len=0
   84 20.885149000 192.168.0.254 → 192.168.0.6  TCP 54 57200 → 80 [FIN, ACK] Seq=2936 Ack=21059 Win=2102272 Len=0
   85 20.885307601  192.168.0.6 → 192.168.0.254 TCP 54 80 → 54793 [FIN, ACK] Seq=1 Ack=2 Win=29312 Len=0
   86 20.885336101  192.168.0.6 → 192.168.0.254 TCP 54 80 → 57200 [FIN, ACK] Seq=21059 Ack=2937 Win=38016 Len=0
   87 20.885631202 192.168.0.254 → 192.168.0.6  TCP 54 57200 → 80 [ACK] Seq=2937 Ack=21060 Win=2102272 Len=0
   88 20.885802603 192.168.0.254 → 192.168.0.6  TCP 54 54793 → 80 [ACK] Seq=2 Ack=2 Win=2102272 Len=0
   89 20.959100397 40.101.147.114 → 192.168.0.6  TCP 54 587 → 37430 [FIN, ACK] Seq=5045 Ack=10068 Win=1048560 Len=0
   90 20.959136097  192.168.0.6 → 40.101.147.114 TCP 54 37430 → 587 [ACK] Seq=10068 Ack=5046 Win=44416 Len=0

上記の通り、TLS1.2でメール送信が行われていることが確認できました。