How to create Partially Connected NNs with prespecified connections using Tensorflow?

I am working on something similar (if not the same thing) in JS right now.

Network structure: The network is a 2d array. Each element in the 2d dimension is an object holding 2 arrays, starts and ends. The starts array holds the outgoing connections. The ends array holds the incoming connections. This way, an end on one node is a reference to a start on another node in the next layer. Each element in starts and ends has 2 variables, weight and value. I've tried setting the value on the node itself (array[x][x].value instead of array[x][y].starts[z].value) but it didn't work out. The value must be carried to the next activation somehow - so I put it directly on the "synapse". (Note that nodes, or neurons, are just conceptual here. I'm just referring to the cell in the array (array[x][y] <- node)).

Connections:

So, I have an "addLink" function that sets the connections. It takes sr, sc, er, ec as arguments (start/end row, start/end column). It creates the "Link" object and places it into two arrays (array[sr][sc].starts and array[er][ec].ends). You would just need an array, or external files... some list of start and end coordinates for each link that you want to form.

My connection solution:

I have implemented "max outgoing connections" for each node, as well as defining "possible connection ranges" for each node. Basically, I define a box size (yMin, yMax, xMin, xMax), it adds the node's coordinates (and inverts for a second box on the other side) to create an array of possible connections and "randomly" chooses a number of nodes from those connections up to max connections. I call it getReachableCells. This allows me to form connections across multiple layers. I also define the probabilities of which connections are chosen (further connections are less likely; one cell to the right is the most likely.) In the future, I plan to (rarely) make one of those reachable cells an input node on another network. I've made a 2d-array utility class that allows me to scale, consolidate, and prune the network.

I plan on creating signalling events that triggers growth and death of nodes and connections. For example, if an input-values array's (x_data) length is greater than the number of input neurons, this will trigger a growth signal. If it happens several times (if the demand is high), the array will be scaled and a new node created. Synapses will have connection strengths that are strengthened with use and weakened with lack of use - eventually being removed if the connection strength hits 0. (I started as a game programmer.)

Also, I made a render class that shows all of the connections as lines on a canvas as well as functions that label inputs and outputs.

The problem that I'm currently facing: Backpropagation. Since I have multiple outputs, I need to adjust the weights associated with each output. I call this an output's "chain". I want to be able to grab the chain during the forward pass so that I don't have to loop over the entire network again for every output to find it, but it's looking like that's the only way. If anyone has any insight on this, please let me know.

In case that you want to connect a placeholder to a new layer you can do as below

x = tf.placeholder(shape=[None, 784], dtype=tf.float32) # mnist for example

x = tf.reshape(x, (-1, 784, 1)) # change to new shape (784 ,1)

x = tf.unstack(self._input,axis=1) # getting a list with 784 elements of 1

con = tf.concat(x[1,2,3,4,5]) # for instance you want only from 1 to 5 inputs/neurons

you can feed your new layer with above con which is only corresponds from the input 1 to 5. you can apply same technique to a layer instead of tf.placeholder

So I've done some more research after posting this and it looks like treating things as separate neural nets and connecting them later in subsequent hidden layers is a good idea. If anyone has other ideas that might be helpful, I'd love to hear, though!