There is few crypto libraries for TLS that works on OpenWrt:

• OpenSSL is a de-facto standard. It's libopenssl takes more than a 1Mb of disk space.
• MbedTLS is a small library developed for embedded devices. Was used by default in OpenWrt before.
• WolfSSL is a small library developed for embedded devices. Supports TLS1.3. Installed by default in OpenWrt 21. But in future this may be changed back to MbedTLS.
• Nettle just a small crypto library without TLS support.
• GnuTLS is GNU project but not often used. Internally uses Nettle.
• LibTomCrypt: used internally in Dropbear SSH daemon. It's not a TLS lib that you may use but all routers have it.

When you are installing some program you may check which library you already have and install a specific version to reuse existing dependency. For example rtty daemon has three versions rtty-mbedtls, rtty-openssl, rtty-wolfssl. Some OpenWrt only packages like kadnode uses only mbedtls and other libraries aren't supported yet.

See also Comparison of TLS implementations