[Kea-users] kea 1.3.0 dying with mysql error

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Kea-users] kea 1.3.0 dying with mysql error

Satish Patel
I have configured kea 1.3.0 with mysql and everything works but after
few hours found kea is dead and spotted following error in log, I have
check everything mysql running fine everything looks good in DB.


2018-06-15 11:48:55.798 INFO  [kea-dhcp4.dhcp4/16020] DHCP4_STARTED
Kea DHCPv4 server version 1.3.0 started
2018-06-15 12:01:29.048 ERROR [kea-dhcp4.dhcpsrv/16020]
DHCPSRV_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable
to execute for <SELECT h.host_id, h.dhcp_identifier,
h.dhcp_identifier_type, h.dhcp4_subnet_id, h.dhcp6_subnet_id,
h.ipv4_address, h.hostname, h.dhcp4_client_classes,
h.dhcp6_client_classes, h.dhcp4_next_server, h.dhcp4_server_hostname,
h.dhcp4_boot_file_name, o.option_id, o.code, o.value,
o.formatted_value, o.space, o.persistent FROM hosts AS h LEFT JOIN
dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id =
? AND h.dhcp_identifier_type = ?    AND h.dhcp_identifier = ? ORDER BY
h.host_id, o.option_id>, reason: MySQL server has gone away (error
code: 2006). Server exiting now!
2018-06-15 12:01:29.055 INFO  [kea-dhcp4.legal-log/16020]
LEGAL_LOG_FILE_CLOSED  @@Missing placeholder %1 for
'/var/log//kea-forensic4.20180615.txt'@@
2018-06-15 12:01:29.056 INFO  [kea-dhcp4.hooks/16020]
HOOKS_LIBRARY_UNLOADED hooks library
/usr/lib64/hooks/libdhcp_legal_log.so successfully unloaded
2018-06-15 12:01:29.056 INFO  [kea-dhcp4.lease_cmds_hooks/16020]
LEASE_CMDS_DEINIT_OK
2018-06-15 12:01:29.056 INFO  [kea-dhcp4.hooks/16020]
HOOKS_LIBRARY_UNLOADED hooks library
/usr/lib64/hooks/libdhcp_lease_cmds.so successfully unloaded
2018-06-15 12:01:29.056 INFO  [kea-dhcp4.host_cmds_hooks/16020]
HOST_CMDS_DEINIT_OK
2018-06-15 12:01:29.056 INFO  [kea-dhcp4.hooks/16020]
HOOKS_LIBRARY_UNLOADED hooks library
/usr/lib64/hooks/libdhcp_host_cmds.so successfully unloaded
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] kea 1.3.0 dying with mysql error

Francis Dupont
Satish Patel writes:
> h.host_id, o.option_id>, reason: MySQL server has gone away (error
> code: 2006). Server exiting now!

=> it is a MySQL error so not a Kea one, but I have 2 extra comments:
 - google gave an idea: increase the maximum packet setting in MySQL
 (I am afraid it won't solve the problem but just try in the case I am wrong)
 
 - Kea 1.4 reconnects databases so the error should be transient
 (vs fatal) in the new 1.4 release.

Regards

Francis Dupont <[hidden email]>
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] kea 1.3.0 dying with mysql error

Tomek Mrugalski
In reply to this post by Satish Patel
On 18/06/2018 01:07, Satish Patel wrote:

> I have configured kea 1.3.0 with mysql and everything works but after
> few hours found kea is dead and spotted following error in log, I have
> check everything mysql running fine everything looks good in DB.
>
> 2018-06-15 11:48:55.798 INFO  [kea-dhcp4.dhcp4/16020] DHCP4_STARTED
> Kea DHCPv4 server version 1.3.0 started
> 2018-06-15 12:01:29.048 ERROR [kea-dhcp4.dhcpsrv/16020]
> DHCPSRV_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable
> to execute for <SELECT h.host_id, h.dhcp_identifier,
> h.dhcp_identifier_type, h.dhcp4_subnet_id, h.dhcp6_subnet_id,
> h.ipv4_address, h.hostname, h.dhcp4_client_classes,
> h.dhcp6_client_classes, h.dhcp4_next_server, h.dhcp4_server_hostname,
> h.dhcp4_boot_file_name, o.option_id, o.code, o.value,
> o.formatted_value, o.space, o.persistent FROM hosts AS h LEFT JOIN
> dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id =
> ? AND h.dhcp_identifier_type = ?    AND h.dhcp_identifier = ? ORDER BY
> h.host_id, o.option_id>, reason: MySQL server has gone away (error
> code: 2006). Server exiting now!
Translation to plain English: MySQL server has gone away (i.e. Kea
connection to MySQL server was terminated for whatever reason, DB
restart, temporary network failure, etc).

In this state Kea would be useless. It would not be able to answer any
packets as it couldn't check lease state, create new leases, extend
existing ones etc. Continuing in this state would give users a false
impression that the server is somehow functional. Therefore kea 1.3
terminates itself as soon as database failure is detected.

This has been improved in 1.4. Kea attempts to reconnect. If this
behavior makes you unhappy, please upgrade.

> 2018-06-15 12:01:29.055 INFO  [kea-dhcp4.legal-log/16020]
> LEGAL_LOG_FILE_CLOSED  @@Missing placeholder %1 for
> '/var/log//kea-forensic4.20180615.txt'@@
This is begign bug in our logging routines. I've submitted bug #5653 for
this. You can view (and possibly comment on it if you like) here:
https://kea.isc.org/ticket/5653

Tomek
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] kea 1.3.0 dying with mysql error

Satish Patel
Thanks you for comments, I told you half story, full story is we have two kea instance for HA and they are pointed to galera DB cluster for centralize Shared DB.

Both node active and inserting records in cluster DB and everything works but every few hour node-2 kea is dying with mentioned error, everything same node dying. Node-1 never had any issue.

Do you think this is clustering issue they both trying to do same thing and one of node dying? I have check network and every possible thing but  nothing wrong there, also I have increase max_packet_size in db but no luck :(

Sent from my iPhone

> On Jun 18, 2018, at 7:29 AM, Tomek Mrugalski <[hidden email]> wrote:
>
>> On 18/06/2018 01:07, Satish Patel wrote:
>> I have configured kea 1.3.0 with mysql and everything works but after
>> few hours found kea is dead and spotted following error in log, I have
>> check everything mysql running fine everything looks good in DB.
>>
>> 2018-06-15 11:48:55.798 INFO  [kea-dhcp4.dhcp4/16020] DHCP4_STARTED
>> Kea DHCPv4 server version 1.3.0 started
>> 2018-06-15 12:01:29.048 ERROR [kea-dhcp4.dhcpsrv/16020]
>> DHCPSRV_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable
>> to execute for <SELECT h.host_id, h.dhcp_identifier,
>> h.dhcp_identifier_type, h.dhcp4_subnet_id, h.dhcp6_subnet_id,
>> h.ipv4_address, h.hostname, h.dhcp4_client_classes,
>> h.dhcp6_client_classes, h.dhcp4_next_server, h.dhcp4_server_hostname,
>> h.dhcp4_boot_file_name, o.option_id, o.code, o.value,
>> o.formatted_value, o.space, o.persistent FROM hosts AS h LEFT JOIN
>> dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id =
>> ? AND h.dhcp_identifier_type = ?    AND h.dhcp_identifier = ? ORDER BY
>> h.host_id, o.option_id>, reason: MySQL server has gone away (error
>> code: 2006). Server exiting now!
> Translation to plain English: MySQL server has gone away (i.e. Kea
> connection to MySQL server was terminated for whatever reason, DB
> restart, temporary network failure, etc).
>
> In this state Kea would be useless. It would not be able to answer any
> packets as it couldn't check lease state, create new leases, extend
> existing ones etc. Continuing in this state would give users a false
> impression that the server is somehow functional. Therefore kea 1.3
> terminates itself as soon as database failure is detected.
>
> This has been improved in 1.4. Kea attempts to reconnect. If this
> behavior makes you unhappy, please upgrade.
>
>> 2018-06-15 12:01:29.055 INFO  [kea-dhcp4.legal-log/16020]
>> LEGAL_LOG_FILE_CLOSED  @@Missing placeholder %1 for
>> '/var/log//kea-forensic4.20180615.txt'@@
> This is begign bug in our logging routines. I've submitted bug #5653 for
> this. You can view (and possibly comment on it if you like) here:
> https://kea.isc.org/ticket/5653
>
> Tomek
> _______________________________________________
> Kea-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/kea-users
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] kea 1.3.0 dying with mysql error

mike-2
On 06/18/2018 05:03 AM, Satish Patel wrote:
> Thanks you for comments, I told you half story, full story is we have two kea instance for HA and they are pointed to galera DB cluster for centralize Shared DB.
>
> Both node active and inserting records in cluster DB and everything works but every few hour node-2 kea is dying with mentioned error, everything same node dying. Node-1 never had any issue.
>
> Do you think this is clustering issue they both trying to do same thing and one of node dying? I have check network and every possible thing but  nothing wrong there, also I have increase max_packet_size in db but no luck :(
>

I have galera cluster too and also observe routine cases of 'server has
gone away' across sql apps, which is being dealt with by application
layer checks to 'reconnect' to the db. Have not investigated the
underlying cause other than tcpdump but I think this is squarely a
galera / mysql issue.

MIke-
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] kea 1.3.0 dying with mysql error

Klaus Steden

This happens to us, too, using a shared MySQL backend for our lease/reservation DBs.

I put Monit up on our Kea backends to automatically restart when it detects that the process isn't running, and with a bit of tweaking, it's been pretty bulletproof, with a bonus of handling other, unrelated crashes (although I haven't really seen any of those).

cheers,
Klaus

On Mon, Jun 18, 2018 at 8:34 AM, <[hidden email]> wrote:
On 06/18/2018 05:03 AM, Satish Patel wrote:
> Thanks you for comments, I told you half story, full story is we have two kea instance for HA and they are pointed to galera DB cluster for centralize Shared DB.
>
> Both node active and inserting records in cluster DB and everything works but every few hour node-2 kea is dying with mentioned error, everything same node dying. Node-1 never had any issue.
>
> Do you think this is clustering issue they both trying to do same thing and one of node dying? I have check network and every possible thing but  nothing wrong there, also I have increase max_packet_size in db but no luck :(
>

I have galera cluster too and also observe routine cases of 'server has
gone away' across sql apps, which is being dealt with by application
layer checks to 'reconnect' to the db. Have not investigated the
underlying cause other than tcpdump but I think this is squarely a
galera / mysql issue.

MIke-
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users


_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users