Prediction: plugin a corelation table (neuron) into a Time-Series Neuron in Keras/ TF

i am adding more details

I have a time series of Babies (1,2,3) showing how many problem they have each week (Born week 1 to week 80) and in which organ (14 organ).

There is a separate numeric time-free correlation tables for babies/organs. showing Xtype blood cells per organ per baby -> Causes problems

My aim is to predict how many problem will NewBaby (Baby4) have, in which organ, per week

Predict this: Week1: 1 problem | Stomach Week1: 1 problem | Liver Week2: 1 problem | Stomach ..

TIME_SERIES_INPUT ——> LSTM (RNN?) ——-
——————————————————————-> MERGE —>
AUXILIARY_INPUTS –> [do something] ———-/*

How can I model this problem with Keras/TF? To be specific firstly should I modify the dataset to include correlation tables. Or can I plug a model into LSTM? I have seen some examples on this board but they are mostly CNN that does not apply to my data set Ps I can insert whole dataset if needed

Thanks

DataSet Looks like 

Week Baby   Age Organ   Date    IssueFound Problem Type
2   Baby2   A0  Organ2  06/24/16    1   F
2   Baby3   A0  Organ11 05/01/17    1   F
2   Baby3   A0  Organ11 05/02/17    1   F
2   Baby3   A0  Organ13 05/03/17    1   F
2   Baby3   A0  Organ13 05/05/17    1   F
3   Baby2   A0  Organ12 06/29/16    1   F
3   Baby2   A0  Organ2  07/01/16    1   F
4   Baby3   A0  Organ13 05/18/17    1   F
4   Baby3   A0  Organ2  05/19/17    1   F

time-free Co relation Table looks like

New X-Type Blood Cell per Organ per baby

Organ   Baby1   Baby2   Baby3   NewBaby
Bladder Organ1  288 247 97  253
Brain   Organ2  2548    1972    992 251
Ears    Organ3  32  254 162 155
Eyes    Organ4  111 0   459 -89
Heart   Organ5  42  1214    0   142
Kidney  Organ6  4   188 186 -87
Kidneys Organ7  806 1764    796 153
Liver   Organ8  658 657 -20 0
Lung    Organ9  0   484 0   0
Nose    Organ10 3328    9940    4435    3223
Pancreas    Organ11 11  389 207 194
Skin    Organ12 664 766 164 -145
Stomach Organ13 246 -71 0   -70
Tongue  Organ14 1403    0   33  0