[GH-ISSUE #75] Error 789 when connection from Windows 10 #64

New issue

Closed

opened 2026-03-02 07:11:27 +03:00 by kerem · 1 comment

kerem commented

2026-03-02 07:11:27 +03:00

Owner

Originally created by @anmiles on GitHub (Apr 26, 2018).
Original GitHub issue: https://github.com/hwdsl2/docker-ipsec-vpn-server/issues/75

Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer.

Docker image run within ubuntu server (EC2).
Security groups allow any connections between my client and server.
Ports 500 and 4500 are open on the server (udp).
PSK is the same on the client and on the server.
Username and password are double-checked.
One-time registry change is applied on windows.
Both IKEEXT and PolicyAgent services are started on windows.

Here is log (client ip address is changed):

Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: packet from 1.2.3.4:500: ignoring unknown Vendor ID payload [01528bbbc00696121849ab9a1c5b2a5100000001]
Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: responding to Main Mode from unknown peer 1.2.3.4 on port 500
Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: Oakley Transform [AES_CBC (256), HMAC_SHA1, DH20] refused
Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: Oakley Transform [AES_CBC (128), HMAC_SHA1, DH19] refused
Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: STATE_MAIN_R1: sent MR1, expecting MI2
Apr 26 07:23:33 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1
Apr 26 07:23:34 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1
Apr 26 07:23:37 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1
Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: deleting incomplete state after 60.000 seconds
Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: deleting state (STATE_MAIN_R1)
Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: deleting connection "l2tp-psk"[1] 1.2.3.4 instance with peer 1.2.3.4 {isakmp=#0/ipsec=#0}

Originally created by @anmiles on GitHub (Apr 26, 2018). Original GitHub issue: https://github.com/hwdsl2/docker-ipsec-vpn-server/issues/75 Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer. Docker image run within ubuntu server (EC2). Security groups allow any connections between my client and server. Ports 500 and 4500 are open on the server (udp). PSK is the same on the client and on the server. Username and password are double-checked. One-time registry change is applied on windows. Both IKEEXT and PolicyAgent services are started on windows. Here is log (client ip address is changed): > Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: packet from 1.2.3.4:500: ignoring unknown Vendor ID payload [01528bbbc00696121849ab9a1c5b2a5100000001] Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: responding to Main Mode from unknown peer 1.2.3.4 on port 500 Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: Oakley Transform [AES_CBC (256), HMAC_SHA1, DH20] refused Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: Oakley Transform [AES_CBC (128), HMAC_SHA1, DH19] refused Apr 26 07:23:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: STATE_MAIN_R1: sent MR1, expecting MI2 Apr 26 07:23:33 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1 Apr 26 07:23:34 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1 Apr 26 07:23:37 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: retransmitting in response to duplicate packet; already STATE_MAIN_R1 Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: deleting incomplete state after 60.000 seconds Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: "l2tp-psk"[1] 1.2.3.4 #1: deleting state (STATE_MAIN_R1) Apr 26 07:24:32 e0de3bb49b45 pluto[2081]: deleting connection "l2tp-psk"[1] 1.2.3.4 instance with peer 1.2.3.4 {isakmp=#0/ipsec=#0}

kerem closed this issue

2026-03-02 07:11:28 +03:00

kerem commented

2026-03-02 07:11:28 +03:00

Author

Owner

@hwdsl2 commented on GitHub (May 14, 2018):

Note: Please first set up your own VPN server.

注：请首先搭建自己的 VPN 服务器。

@anmiles Hello! Your logs indicate that there is some firewall between the VPN server and client which interferes with the connection, or there is a network problem. This is typically not an issue with the VPN server itself.

@hwdsl2 commented on GitHub (May 14, 2018): ### Note: Please first [set up your own VPN server](https://github.com/hwdsl2/setup-ipsec-vpn). ### 注：请首先 [搭建自己的 VPN 服务器](https://github.com/hwdsl2/setup-ipsec-vpn/blob/master/README-zh.md)。 --- @anmiles Hello! Your logs indicate that there is some firewall between the VPN server and client which interferes with the connection, or there is a network problem. This is typically not an issue with the VPN server itself.

No labels

bug

pull-request

question

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/docker-ipsec-vpn-server#64

No description provided.

Rows
Columns

[GH-ISSUE #75] Error 789 when connection from Windows 10 #64

Note: Please first set up your own VPN server.

注：请首先 搭建自己的 VPN 服务器。

注：请首先搭建自己的 VPN 服务器。