[Kea-users] HA lease updates not working after upgrade to 1.8

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Kea-users] HA lease updates not working after upgrade to 1.8

Christian Schuldt
Hey,

I updated one of our Kea hot-standby HA pairs from 1.6 to 1.8 and now HA lease-updates to the standby server no longer work.
The only config change was the required move of the logging section into the dhcp4-server section.

Error on primary:
2020-11-04 13:48:48.754 INFO  [kea-dhcp4.leases/3244.139646677208128] DHCP4_LEASE_ALLOC [hwtype=1 00:01:02:81:41:ed], cid=[no info], tid=0x1ec6987e: lease 192.168.100.211 has been allocated for 86400 seconds
2020-11-04 13:48:48.760 WARN  [kea-dhcp4.ha-hooks/3244.139646677208128] HA_LEASE_UPDATE_FAILED [hwtype=1 00:01:02:81:41:ed], cid=[no info], tid=0x1ec6987e: lease update to ns2-kea (http://10.1.0.254:8080/) failed: unable to update lease for address 192.168.100.211 as it does not exist, error code 1

Error on standby:
2020-11-04 13:48:48.759 ERROR [kea-dhcp4.callouts/15648.140137833094208] HOOKS_CALLOUT_ERROR error returned by callout on hook 1 registered by library with index $lease4_update (callout address 0x7f74577f0d40) (callout duration 1.262 ms)

I already purged both kea and mariadb (leases and reservations) and  thenreinstalled, syncing reservations and then trying again with no change.

I previously tried to manually update a lease on the standby with a query extracted from traffic which also doesn't work. The error seems to originate from the lease_commands hooks:
curl -XPOST http://localhost:8080 -H  "Content-Type: application/json" \
  -d '{ "arguments": { "expire": 1604579347, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "pc.", "hw-address": "00:01:02:73:93:b7", "ip-address": "192.168.210.148", "state": 0, "subnet-id": 124, "valid-lft": 86400 }, "command": "lease4-update", "service": [ "dhcp4" ] }'
[ { "result": 1, "text": "unable to update lease for address 192.168.210.148 as it does not exist" } ]

However when immediately trying "lease4-add" instead of update, it fails with "IPv4 lease already exists."
A manual update also doesn't work on the primary.

When deleting the leases4 table contents on the standby server, initial HA lease synchronization runs without issues, but afterwards updates don't work. Restarting after initial synchronization causes the same lease update errors as during "normal" operation.

The servers configuration is the same (synchronised via git) with the necessary exceptions of interfaces and HA config. Subnets have fixed IDs for database reservations.
HA config only differs in "this-server-name".

Hooks config:

"hooks-libraries": [
  {
    #" lease cmds are required for ha",
    "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
    "parameters": { }
  },
  {
    "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
    "parameters": {
      "high-availability": [
        {
          "this-server-name": "ns2-kea",
          "mode": "hot-standby",
          "heartbeat-delay": 4000,
          "max-response-delay": 15000,
          "max-ack-delay": 5000,
          "max-unacked-clients": 0,
          "sync-page-limit": 5000,
          "sync-timeout": 30000,
          "peers": [
            {
              "name": "ns1-kea",
              "url": "http://10.1.0.253:8080/",
              "role": "primary",
              #"role": "standby",
              "auto-failover": true
            },
            {
              "name": "ns2-kea",
              "url": "http://10.1.0.254:8080/",
              #"role": "primary",
              "role": "standby",
              "auto-failover": true
            }
          ]
        }
      ]
    }
  }
]

Any help would be appreciated.

Best Regards
Christian Schuldt

 
 

   
This email may contain material that is confidential, privileged and/or attorney work product for the sole use of the intended recipient. Any review, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Studio Funk GmbH & Co. KG · Geschäftsführer: Klaus Funk, Markus Weber · Sitz der Gesellschaft: Hamburg · Handelsregister-Nr.: 67553


_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] HA lease updates not working after upgrade to 1.8

Marcin Siodelski
Christian,

Can you please let me know what lease database backend you're currently
using?

Marcin Siodelski
ISC

W dniu 04.11.2020 o 14:24, Christian Schuldt pisze:

> Hey,
>
> I updated one of our Kea hot-standby HA pairs from 1.6 to 1.8 and now HA
> lease-updates to the standby server no longer work.
> The only config change was the required move of the logging section into
> the dhcp4-server section.
>
> Error on primary:
> 2020-11-04 13:48:48.754 INFO  [kea-dhcp4.leases/3244.139646677208128]
> DHCP4_LEASE_ALLOC [hwtype=1 00:01:02:81:41:ed], cid=[no info],
> tid=0x1ec6987e: lease 192.168.100.211 has been allocated for 86400 seconds
> 2020-11-04 13:48:48.760 WARN  [kea-dhcp4.ha-hooks/3244.139646677208128]
> HA_LEASE_UPDATE_FAILED [hwtype=1 00:01:02:81:41:ed], cid=[no info],
> tid=0x1ec6987e: lease update to ns2-kea (http://10.1.0.254:8080/)
> failed: unable to update lease for address 192.168.100.211 as it does
> not exist, error code 1
>
> Error on standby:
> 2020-11-04 13:48:48.759 ERROR [kea-dhcp4.callouts/15648.140137833094208]
> HOOKS_CALLOUT_ERROR error returned by callout on hook 1 registered by
> library with index $lease4_update (callout address 0x7f74577f0d40)
> (callout duration 1.262 ms)
>
> I already purged both kea and mariadb (leases and reservations) and 
> thenreinstalled, syncing reservations and then trying again with no change.
>
> I previously tried to manually update a lease on the standby with a
> query extracted from traffic which also doesn't work. The error seems to
> originate from the lease_commands hooks:
> curl -XPOST http://localhost:8080 -H  "Content-Type: application/json" \
>   -d '{ "arguments": { "expire": 1604579347, "force-create": true,
> "fqdn-fwd": false, "fqdn-rev": false, "hostname": "pc.", "hw-address":
> "00:01:02:73:93:b7", "ip-address": "192.168.210.148", "state": 0,
> "subnet-id": 124, "valid-lft": 86400 }, "command": "lease4-update",
> "service": [ "dhcp4" ] }'
> [ { "result": 1, "text": "unable to update lease for address
> 192.168.210.148 as it does not exist" } ]
>
> However when immediately trying "lease4-add" instead of update, it fails
> with "IPv4 lease already exists."
> A manual update also doesn't work on the primary.
>
> When deleting the leases4 table contents on the standby server, initial
> HA lease synchronization runs without issues, but afterwards updates
> don't work. Restarting after initial synchronization causes the same
> lease update errors as during "normal" operation.
>
> The servers configuration is the same (synchronised via git) with the
> necessary exceptions of interfaces and HA config. Subnets have fixed IDs
> for database reservations.
> HA config only differs in "this-server-name".
>
> Hooks config:
>
> "hooks-libraries": [
>   {
>     #" lease cmds are required for ha",
>     "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
>     "parameters": { }
>   },
>   {
>     "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
>     "parameters": {
>       "high-availability": [
>         {
>           "this-server-name": "ns2-kea",
>           "mode": "hot-standby",
>           "heartbeat-delay": 4000,
>           "max-response-delay": 15000,
>           "max-ack-delay": 5000,
>           "max-unacked-clients": 0,
>           "sync-page-limit": 5000,
>           "sync-timeout": 30000,
>           "peers": [
>             {
>               "name": "ns1-kea",
>               "url": "http://10.1.0.253:8080/",
>               "role": "primary",
>               #"role": "standby",
>               "auto-failover": true
>             },
>             {
>               "name": "ns2-kea",
>               "url": "http://10.1.0.254:8080/",
>               #"role": "primary",
>               "role": "standby",
>               "auto-failover": true
>             }
>           ]
>         }
>       ]
>     }
>   }
> ]
>
> Any help would be appreciated.
>
> Best Regards
> Christian Schuldt
>
>  
>   <http://www.studiofunk.de>
>
>   <https://www.instagram.com/studiofunk/
> <https://www.facebook.com/Studio-Funk-100165310029765/>
> <https://twitter.com/studio_funk>
> This email may contain material that is confidential, privileged and/or
> attorney work product for the sole use of the intended recipient. Any
> review, reliance or distribution by others or forwarding without express
> permission is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
> Studio Funk GmbH & Co. KG · Geschäftsführer: Klaus Funk, Markus Weber ·
> Sitz der Gesellschaft: Hamburg · Handelsregister-Nr.: 67553
>
>
> _______________________________________________
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/kea-users
>

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] HA lease updates not working after upgrade to 1.8

Marcin Siodelski
Apologies, it seems I overlooked the fact that you mentioned MariaDB. We
have recently found the issue in Kea 1.8.0 which causes this behavior
when using High Availability. This issue was corrected in the 1.9.1
version and we will be preparing a maintenance 1.8.1 stable release in a
couple of days. This release is tentatively scheduled for November 12th.

One way would be to try 1.9.1, but this is a development release.
Another way is to wait for a fix to come in 1.8.1.

Marcin Siodelski
ISC

W dniu 04.11.2020 o 16:15, Marcin Siodelski pisze:

> Christian,
>
> Can you please let me know what lease database backend you're currently
> using?
>
> Marcin Siodelski
> ISC
>
> W dniu 04.11.2020 o 14:24, Christian Schuldt pisze:
>> Hey,
>>
>> I updated one of our Kea hot-standby HA pairs from 1.6 to 1.8 and now HA
>> lease-updates to the standby server no longer work.
>> The only config change was the required move of the logging section into
>> the dhcp4-server section.
>>
>> Error on primary:
>> 2020-11-04 13:48:48.754 INFO  [kea-dhcp4.leases/3244.139646677208128]
>> DHCP4_LEASE_ALLOC [hwtype=1 00:01:02:81:41:ed], cid=[no info],
>> tid=0x1ec6987e: lease 192.168.100.211 has been allocated for 86400 seconds
>> 2020-11-04 13:48:48.760 WARN  [kea-dhcp4.ha-hooks/3244.139646677208128]
>> HA_LEASE_UPDATE_FAILED [hwtype=1 00:01:02:81:41:ed], cid=[no info],
>> tid=0x1ec6987e: lease update to ns2-kea (http://10.1.0.254:8080/)
>> failed: unable to update lease for address 192.168.100.211 as it does
>> not exist, error code 1
>>
>> Error on standby:
>> 2020-11-04 13:48:48.759 ERROR [kea-dhcp4.callouts/15648.140137833094208]
>> HOOKS_CALLOUT_ERROR error returned by callout on hook 1 registered by
>> library with index $lease4_update (callout address 0x7f74577f0d40)
>> (callout duration 1.262 ms)
>>
>> I already purged both kea and mariadb (leases and reservations) and 
>> thenreinstalled, syncing reservations and then trying again with no change.
>>
>> I previously tried to manually update a lease on the standby with a
>> query extracted from traffic which also doesn't work. The error seems to
>> originate from the lease_commands hooks:
>> curl -XPOST http://localhost:8080 -H  "Content-Type: application/json" \
>>   -d '{ "arguments": { "expire": 1604579347, "force-create": true,
>> "fqdn-fwd": false, "fqdn-rev": false, "hostname": "pc.", "hw-address":
>> "00:01:02:73:93:b7", "ip-address": "192.168.210.148", "state": 0,
>> "subnet-id": 124, "valid-lft": 86400 }, "command": "lease4-update",
>> "service": [ "dhcp4" ] }'
>> [ { "result": 1, "text": "unable to update lease for address
>> 192.168.210.148 as it does not exist" } ]
>>
>> However when immediately trying "lease4-add" instead of update, it fails
>> with "IPv4 lease already exists."
>> A manual update also doesn't work on the primary.
>>
>> When deleting the leases4 table contents on the standby server, initial
>> HA lease synchronization runs without issues, but afterwards updates
>> don't work. Restarting after initial synchronization causes the same
>> lease update errors as during "normal" operation.
>>
>> The servers configuration is the same (synchronised via git) with the
>> necessary exceptions of interfaces and HA config. Subnets have fixed IDs
>> for database reservations.
>> HA config only differs in "this-server-name".
>>
>> Hooks config:
>>
>> "hooks-libraries": [
>>   {
>>     #" lease cmds are required for ha",
>>     "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
>>     "parameters": { }
>>   },
>>   {
>>     "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
>>     "parameters": {
>>       "high-availability": [
>>         {
>>           "this-server-name": "ns2-kea",
>>           "mode": "hot-standby",
>>           "heartbeat-delay": 4000,
>>           "max-response-delay": 15000,
>>           "max-ack-delay": 5000,
>>           "max-unacked-clients": 0,
>>           "sync-page-limit": 5000,
>>           "sync-timeout": 30000,
>>           "peers": [
>>             {
>>               "name": "ns1-kea",
>>               "url": "http://10.1.0.253:8080/",
>>               "role": "primary",
>>               #"role": "standby",
>>               "auto-failover": true
>>             },
>>             {
>>               "name": "ns2-kea",
>>               "url": "http://10.1.0.254:8080/",
>>               #"role": "primary",
>>               "role": "standby",
>>               "auto-failover": true
>>             }
>>           ]
>>         }
>>       ]
>>     }
>>   }
>> ]
>>
>> Any help would be appreciated.
>>
>> Best Regards
>> Christian Schuldt
>>
>>  
>>   <http://www.studiofunk.de>
>>
>>   <https://www.instagram.com/studiofunk/
>> <https://www.facebook.com/Studio-Funk-100165310029765/>
>> <https://twitter.com/studio_funk>
>> This email may contain material that is confidential, privileged and/or
>> attorney work product for the sole use of the intended recipient. Any
>> review, reliance or distribution by others or forwarding without express
>> permission is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete all copies.
>>
>> Studio Funk GmbH & Co. KG · Geschäftsführer: Klaus Funk, Markus Weber ·
>> Sitz der Gesellschaft: Hamburg · Handelsregister-Nr.: 67553
>>
>>
>> _______________________________________________
>> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>>
>> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>>
>> Kea-users mailing list
>> [hidden email]
>> https://lists.isc.org/mailman/listinfo/kea-users
>>
>
> _______________________________________________
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/kea-users
>

_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users