TransWikia.com

EfficientNet function composition or Hadamard

Data Science Asked by Chris XU on February 15, 2021

In the page 3 of the paper of EfficientNet, there is a equation
$$mathcal{N} = bigodot_{i=1…s} mathcal{F}_{i}^{L_i} big(X_{langle H_i, W_i, C_i rangle}big)$$
where $mathcal{N}$ is the conv net and each $mathcal{F}_i^{L_i}$ is the $i$th-stage layer operator that has length $L_i$.

What I don’t understand is, what is this $odot$ in this equation? Does the author refer to the Hadamard product or does he refer to the function composition? He previously mentioned that $mathcal{N} = mathcal{F}_k odot … odot mathcal{F}_1 (X_1)$, where k is the depth of the net. So I thought it means that $odot$ is just function composition. But EfficientNet has skip connection. In the keras implementation it uses layers.merge.Multiply() so it can also means that the input data $X$ is multiplied with the transformed $mathcal{F}(X)$, and $odot$ maybe means the Hadamard product.

Does anyone knows the answer? Thanks.

One Answer

It probably represents repeated function composition.

Key reasons:

  1. The authors' wording before this equation is: "a list of composed layers".
  2. A Hadamard product could not be applied to layers of different sizes, whereas function composition of course can.
  3. Yes, EfficientNet has skip connections, but I think they try to validate function composition via a distinction between "layers" and "stages". They say that "$F_i^{L_i}$ denotes layer $F_i$ is repeated $L_i$ times in stage $i$". They also explain that "all layers in each stage share the same architecture. Therefore, I think they put skip-connections in the form of function composition, but it is confusing. I think it's their way of compactly expressing the (often) repeated architecture of skip-connected layers.

Correct answer by Benji Albert on February 15, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP