[Kea-users] Remote debugging in KEA

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Kea-users] Remote debugging in KEA

Hreiðar Jóelsson

Hi, can anyone help me or point me in the right direction regarding remote debugging and KEA. I’m developing a hook that queries external Redis database for lease policies and uses them to choose subnet and manipulate DHCP options for incoming requests. My problem is that Kea is crashing in the main process after a pass it a new DHCP option 6 so I need to be able to debug what is going on there. I’m trying to setup remote debug using gdbserver and Jetbrains Clion an I have manage to start a debugging session on my KEA server and connect to it from my development machine but KEA does not run. Any ideas on how this works?

 

Best Regards,
Hreiðar Jóelsson
Network Engineer


_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
lex
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] Remote debugging in KEA

lex
gdb attach should be enough.
Or if you are running keadhcp from start - then just gdb .../keadhcp.. <options>
You can also force system to create core files after crashes

On Fri, Nov 2, 2018, 15:20 Hreiðar Jóelsson <[hidden email] wrote:

Hi, can anyone help me or point me in the right direction regarding remote debugging and KEA. I’m developing a hook that queries external Redis database for lease policies and uses them to choose subnet and manipulate DHCP options for incoming requests. My problem is that Kea is crashing in the main process after a pass it a new DHCP option 6 so I need to be able to debug what is going on there. I’m trying to setup remote debug using gdbserver and Jetbrains Clion an I have manage to start a debugging session on my KEA server and connect to it from my development machine but KEA does not run. Any ideas on how this works?

 

Best Regards,
Hreiðar Jóelsson
Network Engineer

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] Remote debugging in KEA

Hreiðar Jóelsson

thanks, I was not able to get gdb working remotely with Clion but I managed to get some trace of the crash by running it locally on the server. The problem is that I’m not sure what to make of it 😊

 

Thread 1 "kea-dhcp4" received signal SIGSEGV, Segmentation fault.

0x00007ffe4c2338b8 in ?? ()

(gdb) bt

#0  0x00007ffe4c2338b8 in ?? ()

#1  0x00007f5f823fb960 in isc::dhcp::LibDHCP::packOptions4 (buf=..., options=std::multimap with 8 elements = {...}) at libdhcp++.cc:805

#2  0x00007f5f8244acda in isc::dhcp::Pkt4::pack (this=0x557a5c8d2060) at pkt4.cc:138

#3  0x0000557a5b6258f9 in isc::dhcp::Dhcpv4Srv::processPacketPktSend (this=this@entry=0x7ffe4c234960, callout_handle=..., query=..., rsp=...) at dhcp4_srv.cc:1163

#4  0x0000557a5b62ebf9 in isc::dhcp::Dhcpv4Srv::processPacket (this=this@entry=0x7ffe4c234960, query=..., rsp=..., allow_packet_park=allow_packet_park@entry=true) at dhcp4_srv.cc:1110

#5  0x0000557a5b62fd25 in isc::dhcp::Dhcpv4Srv::run_one (this=this@entry=0x7ffe4c234960) at dhcp4_srv.cc:821

#6  0x0000557a5b6302f8 in isc::dhcp::Dhcpv4Srv::run (this=0x7ffe4c234960) at dhcp4_srv.cc:723

#7  0x0000557a5b603466 in main (argc=<optimized out>, argv=<optimized out>) at main.cc:227

 

I ‘we discovered that the server crashes when I feed in DHCP option using isc::asiolink::IOAddress, where the type is ipv4-address, like option 6 – domain-name-servers. Maybe someone can show an example of how one should go about overwriting those kinds of options.

 

Here is more info if someone got some ideas. This is the code that adds the new option to the response object and some log lines just before KEA dies:

 

# add IPAddress option - called from pkt4_send hook

void add4OptionIPAddress(Pkt4Ptr& response, uint8_t opt_code, std::string& opt_value)

    {

        OptionPtr opt = response->getOption(opt_code);

        if(opt)

        {

           response -> delOption(opt_code);

        }

        Option4AddrLst newIpList(opt_code);

        if(opt_value.find(",") != std::string::npos)

        {

                isc::asiolink::IOAddress addr1(opt_value.substr(0,opt_value.find(",")));

                newIpList.addAddress(addr1);

 

                isc::asiolink::IOAddress addr2(opt_value.substr(opt_value.find(",")+1));

                newIpList.addAddress(addr2);

        }

        else

        {

              isc::asiolink::IOAddress addr3(opt_value);

              newIpList.addAddress(addr3);

        }

        opt.reset( &newIpList );

        LOG_DEBUG(my_logger, 50, "dns server list %1").arg(opt->toText());

        LOG_DEBUG(my_logger, 50, "responce befor leave %1 ").arg(response->toText());

        response->addOption(opt);

        LOG_DEBUG(my_logger, 50, "responce after leave %1 ").arg(response->toText());

    }

   

# my_logger last line

2018-11-02 14:43:06.377 DEBUG [kea-dhcp4.my_logger/29189] responce befor leave local_address=83.173.15.80:67, remote_address=10.206.0.2:67, msg_type=DHCPACK (5), transid=0x76fc973b,

options:

  type=001, len=004: 4294967040 (uint32)

  type=003, len=004: 83.173.30.1

  type=006, len=008: 1.1.1.1 8.8.8.8

  type=015, len=013: "gagnaveita.is" (string)

  type=051, len=004: 60 (uint32)

  type=053, len=001: 5 (uint8)

  type=054, len=004: 83.173.15.80

  type=061, len=007: 01:50:a7:2b:ea:cc:e3

 

# kea-dhcp4 last lines

2018-11-02 14:43:06.377 DEBUG [kea-dhcp4.callouts/29189] HOOKS_CALLOUTS_COMPLETE completed callouts for hook pkt4_send (total callouts duration: 0.402 ms)

2018-11-02 14:43:06.377 DEBUG [kea-dhcp4.options/29189] DHCP4_PACKET_PACK [hwtype=1 50:a7:2b:ea:cc:e3], cid=[01:50:a7:2b:ea:cc:e3], tid=0x76fc973b: preparing on-wire format of the packet to be sent

 

 

regards, Hreiðar.

 

From: EZELINK TELECOM <[hidden email]>
Sent: Friday, November 2, 2018 12:46 PM
To: Hreiðar Jóelsson <[hidden email]>
Cc: [hidden email]
Subject: Re: [Kea-users] Remote debugging in KEA

 

gdb attach should be enough.

Or if you are running keadhcp from start - then just gdb .../keadhcp.. <options>

You can also force system to create core files after crashes

On Fri, Nov 2, 2018, 15:20 Hreiðar Jóelsson <[hidden email] wrote:

Hi, can anyone help me or point me in the right direction regarding remote debugging and KEA. I’m developing a hook that queries external Redis database for lease policies and uses them to choose subnet and manipulate DHCP options for incoming requests. My problem is that Kea is crashing in the main process after a pass it a new DHCP option 6 so I need to be able to debug what is going on there. I’m trying to setup remote debug using gdbserver and Jetbrains Clion an I have manage to start a debugging session on my KEA server and connect to it from my development machine but KEA does not run. Any ideas on how this works?

 

Best Regards,
Hreiðar Jóelsson
Network Engineer

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users


_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|

Re: [Kea-users] Remote debugging in KEA

Tomek Mrugalski
On 02/11/2018 15:50, Hreiðar Jóelsson wrote:
> thanks, I was not able to get gdb working remotely with Clion but I
> managed to get some trace of the crash by running it locally on the
> server. The problem is that I’m not sure what to make of it 😊
See some tips below.

> Thread 1 "kea-dhcp4" received signal SIGSEGV, Segmentation fault.
>
> #0  0x00007ffe4c2338b8 in ?? ()
>
> #1  0x00007f5f823fb960 in isc::dhcp::LibDHCP::packOptions4 (buf=...,
> options=std::multimap with 8 elements = {...}) at libdhcp++.cc:805
This is the part of the code that's segfaulting:

    for (OptionCollection::const_iterator it = options.begin();
         it != options.end(); ++it) {

Line 805:            it->second->pack(buf);

This is a packOptions4 method that iterates over DHCP options stored in
Pkt4 packet. It is trying to pack (prepare on-wire representation of the
options). The option collection is a multimap that stores each option
as <unsigned int, OptionPtr>. The first entity is option code, and the
second is supposed to be a smart pointer to Option instance (or a class
derived from Option). It looks like one of the options is not
correctly added to the packet. Since this is the first time such a
problem is reported, I suspect is it your hook that adds the option
in some incorrect way. Can't say what's wrong for sure - perhaps you
added raw pointer, instead of OptionPtr (which is a shared pointer),
maybe it's null or the underlying object is misimplemented.

If I were you, I'd start with printing out it->first before calling.
Something like this added before line 805:

std::cout << "Packing option " << it->first << ", the pointer is " <<
it->second << std::endl;

At the very least, this will print out the code of the option that
causing the crash. Hopefully this will give you some pointers.
If it->second is not null, you may call it->second->toText(0) and print
out the result.

> I ‘we discovered that the server crashes when I feed in DHCP option
> using isc::asiolink::IOAddress, where the type is ipv4-address, like
> option 6 – domain-name-servers. Maybe someone can show an example of
> how one should go about overwriting those kinds of options.
And here's probably the reason. Please take a look at
src/lib/dhcp/tests. There are many examples there. A good file to look
at is option4_addrlist_unittest.cc

Finally, this kind of questions are better suited for kea-dev. Average
users are probably not interested in internal workings of kea.

Speaking of remote debugging, perhaps you want to set Kea running on
your local dev system? You can consider running perfdhcp to generate
client traffic. It's not perfect and has many shortcomings, but it may
be something to look at.

Hope that helps,
Tomek
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users