Testing security of IoT devices

In addition of runtime testing....

You might consider using static program analysis techniques, e.g. with Frama-C or Clang analyzer. Read also this draft report for more, and consider using Bismon after september 2020. See also Chariot and Decoder and Vessedia European projects.

Another approach would be a metaprogramming approach : you would write (or use) your own C code generator from some higher level domain specific language (perhaps of your invention, but then read the Dragon book and Programming Language Pragmatics). This has been used in SWIG and you could use tools like Coq to prove some safety properties (e.g. of your C code generator, or sometimes and partly of the generated C code)

If you compile your IoT code with a recent GCC compiler, consider writing your own GCC plugin to check some safety properties. Be of course aware of Rice's theorem (and see also the ambitious RefPerSys project).

Look also inside the CompCert project.

This really depends on how you are connecting to the IoT network in the first place. I am assuming you mean via TCP/IP.

Depending on how far you wish to go with this, you may want to look into a bootable Kali Linux USB to run the pentests. (Its literally what that OS was designed to do)

You can use Nmap in linux/Windows to determine ports that are open/answering on the LAN. (this will expose services that are answering to client probes.) it comes pre-installed in kali.

Once you have found open services/ports you can use something like metasploit to try and exploit the vulnerabilities your devices may or may not have.

unfortunately explaining how to use these programs or operating systems is outside the scope of this question. I hope it points you in the right direction.