[Kea-users] DHCP4_PACKET_SEND_FAIL sendmsg() returned with an error: No route to host

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Kea-users] DHCP4_PACKET_SEND_FAIL sendmsg() returned with an error: No route to host

Brian Hechinger
I'm trying to setup Kea here and I'm having a bit of trouble.


Setup is a Cisco Nexus 3132Q-V setup as a DHCP relay shipping requests
off to Kea. Kea is configured to use a pgsql database.


Kea server: 10.42.1.11

Cisco: 10.42.3.1 (Also 10.42.1.1 as it's the core)


 From the attached log you can see that Kea receives the request but
fails to send the response claiming no route to host. It's trying to
send the response back to the relay which I can in fact ping from the
kea server. Watching packet capture on the switch it doesn't even
attempt to send the packet at all.


What am I missing here that it can't send the response?


[root@dhcp1 log]# ping -c 1 10.42.3.1
PING 10.42.3.1 (10.42.3.1) 56(84) bytes of data.
64 bytes from 10.42.3.1: icmp_seq=1 ttl=255 time=0.343 ms

--- 10.42.3.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.343/0.343/0.343/0.000 ms
[root@dhcp1 log]#


2020-05-11 19:54:00.344 DEBUG [kea-dhcp4.packets/12756]
DHCP4_BUFFER_RECEIVED received buffer from 10.42.3.1:67 to 10.42.1.11:67
over interface dhcp1
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.options/12756]
DHCP4_BUFFER_UNPACK parsing buffer received from 10.42.3.1 to 10.42.1.11
over interface dhcp1
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.packets/12756]
DHCP4_PACKET_RECEIVED [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: DHCPDISCOVER (type 1)
received from 10.42.3.1 to 10.42.1.11 on interface dhcp1
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.packets/12756] DHCP4_QUERY_DATA
[hwtype=1 a8:20:66:25:d5:ca], cid=[01:a8:20:66:25:d5:ca],
tid=0xbff6ccb9, packet details: local_address=10.42.1.11:67,
remote_address=10.42.3.1:67, msg_type=DHCPDISCOVER (1), transid=0xbff6ccb9,
options:
   type=012, len=014: "BrianHechinger" (string)
   type=051, len=004: 7776000 (uint32)
   type=053, len=001: 1 (uint8)
   type=055, len=010: 1(uint8) 121(uint8) 3(uint8) 6(uint8) 15(uint8)
119(uint8) 252(uint8) 95(uint8) 44(uint8) 46(uint8)
   type=057, len=002: 1500 (uint16)
   type=061, len=007: 01:a8:20:66:25:d5:ca
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.dhcpsrv/12756]
DHCPSRV_CFGMGR_SUBNET4_ADDR selected subnet 10.42.3.0/24 for packet
received by matching address 10.42.3.1
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.packets/12756]
DHCP4_SUBNET_SELECTED [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: the subnet with ID 423 was
selected for client assignments
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.packets/12756]
DHCP4_SUBNET_DATA [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: the selected subnet details:
10.42.3.0/24
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_IDENTIFIER get one host with IPv4
reservation for subnet id 423, identified by hwaddr=A8206625D5CA
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_IDENTIFIER get all hosts with reservations using
identifier: hwaddr=A8206625D5CA
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_IDENTIFIER_COUNT using identifier hwaddr=A8206625D5CA,
found 0 host(s)
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_IDENTIFIER_NULL host not found using subnet
id 423 and identifier hwaddr=A8206625D5CA
2020-05-11 19:54:00.345 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_MGR_ALTERNATE_GET4_SUBNET_ID_IDENTIFIER get one host with IPv4
reservation for subnet id 423, identified by hwaddr=A8206625D5CA
2020-05-11 19:54:00.349 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_MGR_ALTERNATE_GET4_SUBNET_ID_IDENTIFIER_NULL host not found using
subnet id 423 and identifier hwaddr=A8206625D5CA
2020-05-11 19:54:00.349 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_IDENTIFIER get one host with IPv4
reservation for subnet id 423, identified by client-id=01A8206625D5CA
2020-05-11 19:54:00.350 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_IDENTIFIER get all hosts with reservations using
identifier: client-id=01A8206625D5CA
2020-05-11 19:54:00.350 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_IDENTIFIER_COUNT using identifier
client-id=01A8206625D5CA, found 0 host(s)
2020-05-11 19:54:00.350 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_IDENTIFIER_NULL host not found using subnet
id 423 and identifier client-id=01A8206625D5CA
2020-05-11 19:54:00.350 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_MGR_ALTERNATE_GET4_SUBNET_ID_IDENTIFIER get one host with IPv4
reservation for subnet id 423, identified by client-id=01A8206625D5CA
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_MGR_ALTERNATE_GET4_SUBNET_ID_IDENTIFIER_NULL host not found using
subnet id 423 and identifier client-id=01A8206625D5CA
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.dhcp4/12756]
DHCP4_CLASS_ASSIGNED [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: client packet has been
assigned to the following class(es): UNKNOWN
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.dhcp4/12756]
DHCP4_CLASS_ASSIGNED [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: client packet has been
assigned to the following class(es): ALL, UNKNOWN
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.ddns/12756]
DHCP4_CLIENT_HOSTNAME_PROCESS [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: processing client's Hostname
option
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.ddns/12756]
DHCP4_CLIENT_HOSTNAME_DATA [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: client sent Hostname option:
BrianHechinger
2020-05-11 19:54:00.352 DEBUG [kea-dhcp4.ddns/12756]
DHCP4_CLIENT_HOSTNAME_DATA [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: client sent Hostname option:
BrianHechinger
2020-05-11 19:54:00.353 DEBUG [kea-dhcp4.ddns/12756]
DHCP4_RESPONSE_HOSTNAME_DATA [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: including Hostname option in
the server's response: brianhechinger
2020-05-11 19:54:00.353 DEBUG [kea-dhcp4.dhcpsrv/12756]
DHCPSRV_PGSQL_GET_CLIENTID obtaining IPv4 leases for client ID
01:a8:20:66:25:d5:ca
2020-05-11 19:54:00.354 DEBUG [kea-dhcp4.dhcpsrv/12756]
DHCPSRV_PGSQL_GET_HWADDR obtaining IPv4 leases for hardware address
hwtype=1 a8:20:66:25:d5:ca
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.alloc-engine/12756]
ALLOC_ENGINE_V4_OFFER_NEW_LEASE allocation engine will try to offer new
lease to the client [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_ADDRESS4 get one host with reservation for
subnet id 423 and IPv4 address 10.42.3.137
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_ADDRESS4 get all hosts with reservations for IPv4
address 10.42.3.137
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ALL_ADDRESS4_COUNT using address 10.42.3.137, found 0 host(s)
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_CFG_GET_ONE_SUBNET_ID_ADDRESS4_NULL host not found using subnet id
423 and address 10.42.3.137
2020-05-11 19:54:00.357 DEBUG [kea-dhcp4.hosts/12756]
HOSTS_MGR_ALTERNATE_GET4_SUBNET_ID_ADDRESS4 trying alternate sources for
host using subnet id 423 and address 10.42.3.137
2020-05-11 19:54:00.360 DEBUG [kea-dhcp4.dhcpsrv/12756]
DHCPSRV_PGSQL_GET_ADDR4 obtaining IPv4 lease for address 10.42.3.137
2020-05-11 19:54:00.362 DEBUG [kea-dhcp4.dhcpsrv/12756]
DHCPSRV_PGSQL_GET_ADDR4 obtaining IPv4 lease for address 10.42.3.137
2020-05-11 19:54:00.363 INFO  [kea-dhcp4.leases/12756]
DHCP4_LEASE_ADVERT [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: lease 10.42.3.137 will be
advertised
2020-05-11 19:54:00.363 DEBUG [kea-dhcp4.options/12756]
DHCP4_PACKET_PACK [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: preparing on-wire format of
the packet to be sent
2020-05-11 19:54:00.363 DEBUG [kea-dhcp4.packets/12756]
DHCP4_PACKET_SEND [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: trying to send packet
DHCPOFFER (type 2) from 10.42.1.11:67 to 10.42.3.1:67 on interface dhcp1
2020-05-11 19:54:00.363 DEBUG [kea-dhcp4.packets/12756]
DHCP4_RESPONSE_DATA [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: responding with packet
DHCPOFFER (type 2), packet details: local_address=10.42.1.11:67,
remote_address=10.42.3.1:67, msg_type=DHCPOFFER (2), transid=0xbff6ccb9,
options:
   type=001, len=004: 4294967040 (uint32)
   type=003, len=004: 10.42.1.1
   type=006, len=008: 8.8.8.8 8.8.4.4
   type=012, len=014: "brianhechinger" (string)
   type=015, len=012: "4amlunch.net" (string)
   type=051, len=004: 4000 (uint32)
   type=053, len=001: 2 (uint8)
   type=054, len=004: 10.42.1.11
   type=058, len=004: 2000 (uint32)
   type=059, len=004: 3500 (uint32)
   type=061, len=007: 01:a8:20:66:25:d5:ca
   type=119, len=043: "4amlunch.net." (fqdn)
"wonko.home.cns.4amlunch.net." (fqdn)
2020-05-11 19:54:00.364 ERROR [kea-dhcp4.packets/12756]
DHCP4_PACKET_SEND_FAIL [hwtype=1 a8:20:66:25:d5:ca],
cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: failed to send DHCPv4
packet: pkt4 send failed: sendmsg() returned  with an error: No route to
host

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] DHCP4_PACKET_SEND_FAIL sendmsg() returned with an error: No route to host

Joshua Schaeffer


On 5/11/20 2:04 PM, Brian Hechinger wrote:
Watching packet capture on the switch it doesn't even attempt to send the packet at all.
Do you mean that you can see the packet show up on the switch interface but it never gets forwarded, or that the packet never gets to the switch in the first place?
2020-05-11 19:54:00.364 ERROR [kea-dhcp4.packets/12756] DHCP4_PACKET_SEND_FAIL [hwtype=1 a8:20:66:25:d5:ca], cid=[01:a8:20:66:25:d5:ca], tid=0xbff6ccb9: failed to send DHCPv4 packet: pkt4 send failed: sendmsg() returned  with an error: No route to host
That's curious. Perhaps you could provide some more troubleshooting information?
- Do you have multiple interfaces on the DHCP server or multiple routing tables? `ip rule list`
- What does your routing table show? `ip route show` and `ip route get 10.42.3.1`
- What does a traceroute show? `traceroute 10.42.3.1`
- How is Kea's interfaces-config map configured in kea-dhcp4.conf?
-- 
Thanks,
Joshua Schaeffer

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] DHCP4_PACKET_SEND_FAIL sendmsg() returned with an error: No route to host

Joshua Schaeffer


On 5/11/20 4:10 PM, Brian Hechinger wrote:

It never gets to the switch in the first place. Watching tcpdump on the dhcp server I only see the requests coming in but never sending anything out.

That's what I figured but wanted to be sure.

Traceroute doesn't work oddly enough. It just times out.

That can be typical in some networks as a lot of firewalls drop ICMP packets by default.
- How is Kea's interfaces-config map configured in kea-dhcp4.conf?

    "interfaces-config": {
        "interfaces": [ "*" ],
        "dhcp-socket-type": "udp"
    },

AFAIK everything looks fine. Can't give you an definitive answer as to why when Kea sends a packet the OS thinks the host has no route:

- Could be a firewall or security context issue. Check IPTables, AppArmor, SELinux, etc.
- As a test you could try changing to raw sockets (remove "dhcp-socket-type": "udp"). If that does work I'm not sure why it would as it's a relayed request.
- Force Kea to use the kernel routing table by adding  "outbound-interface": "use-routing" to the configuration (after re-adding "dhcp-socket-type": "udp" back to the config).
-- 
Thanks,
Joshua Schaeffer

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] DHCP4_PACKET_SEND_FAIL sendmsg() returned with an error: No route to host

Brian Hechinger
On 5/11/2020 8:25 PM, Joshua Schaeffer wrote:
On 5/11/20 4:10 PM, Brian Hechinger wrote:

It never gets to the switch in the first place. Watching tcpdump on the dhcp server I only see the requests coming in but never sending anything out.

That's what I figured but wanted to be sure.

Traceroute doesn't work oddly enough. It just times out.

That can be typical in some networks as a lot of firewalls drop ICMP packets by default.


There is no firewall in place though. packet path for this traceroute should be: vm -> switch -> other interface on same switch.


Something doesn't seem quite right here.


- How is Kea's interfaces-config map configured in kea-dhcp4.conf?

    "interfaces-config": {
        "interfaces": [ "*" ],
        "dhcp-socket-type": "udp"
    },

AFAIK everything looks fine. Can't give you an definitive answer as to why when Kea sends a packet the OS thinks the host has no route:

- Could be a firewall or security context issue. Check IPTables, AppArmor, SELinux, etc.


None of those are running, no.


- As a test you could try changing to raw sockets (remove "dhcp-socket-type": "udp"). If that does work I'm not sure why it would as it's a relayed request.


Didn't make a different.


- Force Kea to use the kernel routing table by adding  "outbound-interface": "use-routing" to the configuration (after re-adding "dhcp-socket-type": "udp" back to the config).


Now it silently fails. Kea *thinks* it sent the packet but it never goes anywhere.


I'm going to dig into this weird networking that's going on here because I'm starting to think it's definitely not Kea who is to blame here.


-brian


_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users