Gaussian fit with asymmetric background

I have to fit this peak (which seems to be a double gaussian from other experiments) to find the peak position of the one at around 2.43 eV.

    data={{2.06667, 0.215727}, {2.06839, 0.215529}, {2.07012, 
  0.21555}, {2.07185, 0.215398}, {2.07358, 0.215041}, {2.07531, 
  0.215536}, {2.07705, 0.215266}, {2.07879, 0.215139}, {2.08054, 
  0.215175}, {2.08228, 0.215206}, {2.08403, 0.215189}, {2.08579, 
  0.214647}, {2.08754, 0.215197}, {2.0893, 0.214909}, {2.09106, 
  0.214751}, {2.09283, 0.214884}, {2.09459, 0.215019}, {2.09637, 
  0.214825}, {2.09814, 0.214692}, {2.09992, 0.214428}, {2.10169, 
  0.214842}, {2.10348, 0.214808}, {2.10526, 0.214428}, {2.10705, 
  0.214407}, {2.10884, 0.214032}, {2.11064, 0.214027}, {2.11244, 
  0.214008}, {2.11424, 0.214177}, {2.11604, 0.21424}, {2.11785, 
  0.214532}, {2.11966, 0.214299}, {2.12147, 0.214316}, {2.12329, 
  0.21391}, {2.12511, 0.214477}, {2.12693, 0.214257}, {2.12876, 
  0.213797}, {2.13058, 0.213738}, {2.13242, 0.213865}, {2.13425, 
  0.213935}, {2.13609, 0.213798}, {2.13793, 0.213884}, {2.13978, 
  0.213478}, {2.14162, 0.213657}, {2.14347, 0.213874}, {2.14533, 
  0.21392}, {2.14719, 0.213618}, {2.14905, 0.213175}, {2.15091, 
  0.213358}, {2.15278, 0.213183}, {2.15465, 0.213233}, {2.15652, 
  0.213574}, {2.1584, 0.213447}, {2.16028, 0.213332}, {2.16216, 
  0.212792}, {2.16405, 0.213083}, {2.16594, 0.213276}, {2.16783, 
  0.213123}, {2.16973, 0.212651}, {2.17163, 0.212897}, {2.17353, 
  0.212469}, {2.17544, 0.212411}, {2.17735, 0.212462}, {2.17926, 
  0.212698}, {2.18118, 0.212527}, {2.1831, 0.212479}, {2.18502, 
  0.2122}, {2.18695, 0.212427}, {2.18888, 0.212564}, {2.19081, 
  0.212799}, {2.19275, 0.212312}, {2.19469, 0.212239}, {2.19663, 
  0.212239}, {2.19858, 0.212275}, {2.20053, 0.212285}, {2.20249, 
  0.211835}, {2.20444, 0.212048}, {2.20641, 0.212331}, {2.20837, 
  0.212339}, {2.21034, 0.211921}, {2.21231, 0.212142}, {2.21429, 
  0.211866}, {2.21626, 0.211976}, {2.21825, 0.212079}, {2.22023, 
  0.212176}, {2.22222, 0.212183}, {2.22422, 0.211872}, {2.22621, 
  0.211987}, {2.22821, 0.211692}, {2.23022, 0.211509}, {2.23222, 
  0.211677}, {2.23423, 0.211913}, {2.23625, 0.211844}, {2.23827, 
  0.211869}, {2.24029, 0.211999}, {2.24231, 0.21177}, {2.24434, 
  0.21181}, {2.24638, 0.211799}, {2.24841, 0.2115}, {2.25045, 
  0.211384}, {2.2525, 0.211125}, {2.25455, 0.211168}, {2.2566, 
  0.211482}, {2.25865, 0.211696}, {2.26071, 0.211474}, {2.26277, 
  0.211495}, {2.26484, 0.211372}, {2.26691, 0.211378}, {2.26898, 
  0.211583}, {2.27106, 0.211829}, {2.27314, 0.21167}, {2.27523, 
  0.211629}, {2.27732, 0.211505}, {2.27941, 0.211581}, {2.28151, 
  0.211718}, {2.28361, 0.211762}, {2.28571, 0.211559}, {2.28782, 
  0.211443}, {2.28994, 0.211538}, {2.29205, 0.211743}, {2.29417, 
  0.211995}, {2.2963, 0.212222}, {2.29842, 0.212371}, {2.30056, 
  0.21248}, {2.30269, 0.21238}, {2.30483, 0.212433}, {2.30698, 
  0.212817}, {2.30912, 0.213346}, {2.31128, 0.213584}, {2.31343, 
  0.213824}, {2.31559, 0.213913}, {2.31776, 0.214013}, {2.31993, 
  0.214608}, {2.3221, 0.21533}, {2.32427, 0.215846}, {2.32645, 
  0.216316}, {2.32864, 0.216903}, {2.33083, 0.217651}, {2.33302, 
  0.218499}, {2.33522, 0.21934}, {2.33742, 0.22047}, {2.33962, 
  0.221687}, {2.34183, 0.223155}, {2.34405, 0.224991}, {2.34626, 
  0.227156}, {2.34848, 0.229401}, {2.35071, 0.231942}, {2.35294, 
  0.235142}, {2.35518, 0.23886}, {2.35741, 0.243024}, {2.35966, 
  0.247899}, {2.3619, 0.253735}, {2.36416, 0.26046}, {2.36641, 
  0.268114}, {2.36867, 0.276732}, {2.37094, 0.286951}, {2.37321, 
  0.29866}, {2.37548, 0.311837}, {2.37776, 0.326513}, {2.38004, 
  0.342403}, {2.38232, 0.359909}, {2.38462, 0.379028}, {2.38691, 
  0.398979}, {2.38921, 0.419082}, {2.39151, 0.439559}, {2.39382, 
  0.459828}, {2.39614, 0.479065}, {2.39845, 0.497439}, {2.40077, 
  0.514591}, {2.4031, 0.529937}, {2.40543, 0.543218}, {2.40777, 
  0.554509}, {2.41011, 0.564041}, {2.41245, 0.571543}, {2.4148, 
  0.577318}, {2.41715, 0.581664}, {2.41951, 0.584684}, {2.42188, 
  0.586367}, {2.42424, 0.586628}, {2.42661, 0.586055}, {2.42899, 
  0.585033}, {2.43137, 0.583433}, {2.43376, 0.580649}, {2.43615, 
  0.577098}, {2.43854, 0.573302}, {2.44094, 0.56914}, {2.44335, 
  0.564739}, {2.44576, 0.559639}, {2.44817, 0.554195}, {2.45059, 
  0.548528}, {2.45302, 0.542444}, {2.45545, 0.535958}, {2.45788, 
  0.529288}, {2.46032, 0.522394}, {2.46276, 0.515344}, {2.46521, 
  0.50772}, {2.46766, 0.499923}, {2.47012, 0.492349}, {2.47258, 
  0.484649}, {2.47505, 0.476646}, {2.47752, 0.469003}, {2.48, 
  0.461086}, {2.48248, 0.452893}, {2.48497, 0.445074}, {2.48746, 
  0.437535}, {2.48996, 0.430001}, {2.49246, 0.422614}, {2.49497, 
  0.415417}, {2.49748, 0.408381}, {2.5, 0.401691}, {2.50252, 
  0.395507}, {2.50505, 0.389155}, {2.50758, 0.383317}, {2.51012, 
  0.377561}, {2.51266, 0.372162}, {2.51521, 0.367216}, {2.51777, 
  0.362904}, {2.52033, 0.358221}, {2.52289, 0.354108}, {2.52546, 
  0.349664}, {2.52803, 0.34549}, {2.53061, 0.342175}, {2.5332, 
  0.338932}, {2.53579, 0.335653}, {2.53838, 0.332509}, {2.54098, 
  0.330032}, {2.54359, 0.327278}, {2.5462, 0.324634}, {2.54882, 
  0.322354}, {2.55144, 0.320283}, {2.55407, 0.318381}, {2.5567, 
  0.316944}, {2.55934, 0.315025}, {2.56198, 0.313586}, {2.56463, 
  0.312301}, {2.56729, 0.310847}, {2.56995, 0.309566}, {2.57261, 
  0.308136}, {2.57529, 0.307526}, {2.57796, 0.306777}, {2.58065, 
  0.306077}, {2.58333, 0.305302}, {2.58603, 0.304558}, {2.58873, 
  0.304452}, {2.59143, 0.303739}, {2.59414, 0.303006}, {2.59686, 
  0.302941}, {2.59958, 0.302988}, {2.60231, 0.303308}, {2.60504, 
  0.303066}, {2.60778, 0.302764}, {2.61053, 0.302531}, {2.61328, 
  0.302819}, {2.61603, 0.302885}, {2.6188, 0.303437}, {2.62156, 
  0.303408}, {2.62434, 0.303414}, {2.62712, 0.303846}, {2.6299, 
  0.303887}, {2.6327, 0.304515}, {2.63549, 0.304799}, {2.6383, 
  0.305556}, {2.64111, 0.305583}, {2.64392, 0.305715}, {2.64674, 
  0.30666}, {2.64957, 0.30706}, {2.65241, 0.307935}, {2.65525, 
  0.308014}, {2.65809, 0.308626}, {2.66094, 0.309271}, {2.6638, 
  0.309877}, {2.66667, 0.310236}, {2.66954, 0.310935}, {2.67241, 
  0.311452}, {2.6753, 0.311984}, {2.67819, 0.312729}, {2.68108, 
  0.313048}, {2.68398, 0.313644}, {2.68689, 0.314562}, {2.6898, 
  0.315255}, {2.69273, 0.315786}, {2.69565, 0.316276}, {2.69859, 
  0.316942}, {2.70153, 0.317694}, {2.70447, 0.318433}, {2.70742, 
  0.318904}, {2.71038, 0.3197}, {2.71335, 0.320455}, {2.71632, 
  0.321375}, {2.7193, 0.322147}, {2.72228, 0.322897}, {2.72527, 
  0.323881}, {2.72827, 0.324675}, {2.73128, 0.325227}, {2.73429, 
  0.326094}, {2.73731, 0.326372}, {2.74033, 0.327988}, {2.74336, 
  0.328584}, {2.7464, 0.329412}, {2.74945, 0.330486}, {2.7525, 
  0.33083}, {2.75556, 0.33201}, {2.75862, 0.332511}, {2.76169, 
  0.334126}, {2.76477, 0.334961}, {2.76786, 0.336079}, {2.77095, 
  0.3368}, {2.77405, 0.337791}, {2.77716, 0.338879}, {2.78027, 
  0.340155}, {2.78339, 0.34079}, {2.78652, 0.341493}, {2.78965, 
  0.343277}, {2.79279, 0.34403}, {2.79594, 0.34526}, {2.7991, 
  0.346692}, {2.80226, 0.348126}, {2.80543, 0.348708}, {2.80861, 
  0.349901}, {2.81179, 0.350647}, {2.81498, 0.351894}, {2.81818, 
  0.353058}, {2.82139, 0.354011}, {2.8246, 0.355692}, {2.82782, 
  0.35666}, {2.83105, 0.357611}, {2.83429, 0.358476}, {2.83753, 
  0.360131}, {2.84078, 0.361664}, {2.84404, 0.362744}, {2.8473, 
  0.363835}, {2.85057, 0.365215}, {2.85386, 0.366189}, {2.85714, 
  0.367669}, {2.86044, 0.36858}, {2.86374, 0.370676}, {2.86705, 
  0.371305}, {2.87037, 0.372348}, {2.8737, 0.374002}, {2.87703, 
  0.37555}, {2.88037, 0.376841}, {2.88372, 0.378641}, {2.88708, 
  0.380673}, {2.89044, 0.381399}, {2.89382, 0.382739}, {2.8972, 
  0.384623}, {2.90058, 0.386228}, {2.90398, 0.387827}, {2.90739, 
  0.38958}, {2.9108, 0.39039}, {2.91422, 0.392615}, {2.91765, 
  0.394514}, {2.92108, 0.396006}, {2.92453, 0.397428}, {2.92798, 
  0.399816}, {2.93144, 0.401183}, {2.93491, 0.402855}, {2.93839, 
  0.405623}, {2.94187, 0.406678}, {2.94537, 0.408336}, {2.94887, 
  0.410715}, {2.95238, 0.413208}, {2.9559, 0.414921}, {2.95943, 
  0.416709}, {2.96296, 0.418627}, {2.96651, 0.421141}, {2.97006, 
  0.423476}, {2.97362, 0.426023}, {2.97719, 0.427544}, {2.98077, 
  0.430291}, {2.98436, 0.432362}, {2.98795, 0.434549}, {2.99156, 
  0.436489}, {2.99517, 0.437908}, {2.99879, 0.441044}, {3.00242, 
  0.443634}, {3.00606, 0.446021}, {3.00971, 0.449037}, {3.01337, 
  0.450737}, {3.01703, 0.453765}, {3.02071, 0.456482}, {3.02439, 
  0.458964}, {3.02808, 0.462196}, {3.03178, 0.464811}, {3.0355, 
  0.468054}, {3.03922, 0.469916}, {3.04294, 0.473378}, {3.04668, 
  0.4771}, {3.05043, 0.480509}, {3.05419, 0.483754}, {3.05795, 
  0.48658}, {3.06173, 0.489552}, {3.06551, 0.493718}, {3.06931, 
  0.496792}, {3.07311, 0.500654}, {3.07692, 0.504402}, {3.08075, 
  0.50924}, {3.08458, 0.51352}, {3.08842, 0.517255}, {3.09227, 
  0.521481}, {3.09613, 0.525424}, {3.1, 0.530235}, {3.10388, 
  0.535117}, {3.10777, 0.539075}, {3.11167, 0.543361}, {3.11558, 
  0.547666}, {3.1195, 0.551461}, {3.12343, 0.555137}, {3.12736, 
  0.558164}, {3.13131, 0.562037}, {3.13527, 0.565307}, {3.13924, 
  0.567252}, {3.14322, 0.56873}, {3.14721, 0.5714}, {3.15121, 
  0.573256}, {3.15522, 0.574469}, {3.15924, 0.576343}, {3.16327, 
  0.577993}, {3.16731, 0.57796}, {3.17136, 0.57948}, {3.17542, 
  0.581357}, {3.17949, 0.581841}, {3.18357, 0.582884}, {3.18766, 
  0.583556}, {3.19176, 0.583867}, {3.19588, 0.585938}, {3.2, 
  0.587387}, {3.20413, 0.588106}, {3.20828, 0.589434}, {3.21244, 
  0.590415}, {3.2166, 0.591923}, {3.22078, 0.593453}, {3.22497, 
  0.594097}, {3.22917, 0.595748}, {3.23338, 0.597168}, {3.2376, 
  0.599144}, {3.24183, 0.600091}, {3.24607, 0.600331}, {3.25033, 
  0.601975}, {3.25459, 0.603644}, {3.25887, 0.605401}, {3.26316, 
  0.606527}, {3.26746, 0.606616}, {3.27177, 0.609499}, {3.27609, 
  0.614912}, {3.28042, 0.616802}, {3.28477, 0.616436}, {3.28912, 
  0.618491}, {3.29349, 0.61805}, {3.29787, 0.616517}, {3.30226, 
  0.617677}, {3.30667, 0.619628}, {3.31108, 0.620627}, {3.31551, 
  0.621646}, {3.31995, 0.622191}, {3.3244, 0.62302}, {3.32886, 
  0.624218}, {3.33333, 0.623998}, {3.33782, 0.624818}, {3.34232, 
  0.626694}, {3.34683, 0.627009}, {3.35135, 0.628262}, {3.35589, 
  0.629078}, {3.36043, 0.629209}, {3.36499, 0.629994}, {3.36957, 
  0.631635}, {3.37415, 0.633532}, {3.37875, 0.634681}, {3.38336, 
  0.63368}, {3.38798, 0.634317}, {3.39261, 0.636057}, {3.39726, 
  0.636847}, {3.40192, 0.638688}, {3.40659, 0.640597}, {3.41128, 
  0.639873}, {3.41598, 0.640696}, {3.42069, 0.643355}, {3.42541, 
  0.645152}, {3.43015, 0.645586}, {3.4349, 0.645957}, {3.43967, 
  0.646863}, {3.44444, 0.6478}, {3.44924, 0.649076}, {3.45404, 
  0.650161}, {3.45886, 0.650472}, {3.46369, 0.651685}, {3.46853, 
  0.653338}, {3.47339, 0.654356}, {3.47826, 0.655443}, {3.48315, 
  0.656204}, {3.48805, 0.656692}, {3.49296, 0.657204}, {3.49788, 
  0.658928}, {3.50282, 0.660147}, {3.50778, 0.660302}, {3.51275, 
  0.661486}, {3.51773, 0.662766}, {3.52273, 0.66368}, {3.52774, 
  0.664209}, {3.53276, 0.66437}, {3.5378, 0.666323}, {3.54286, 
  0.667253}}

Initially I thought was pretty straightforward. Here is the code I am using

NonlinearModelFit[
  data[[100 ;; 330]], {amp1*Exp[-(x - mu1)^2/2 s1^2] + d2 + 
    amp2*Exp[-(x - mu2)^2/2 s2^2] - m*x + d, 
   mu1 < mu2}, {{s1, 20}, {amp1, 0.23}, {mu1, 2.43}, {m, 0.1}, {d, 
    0.3}, {amp2, 0.2}, {mu2, 2.44}, {s2, 40}, {d2, 0.2}}, x, 
  MaxIterations -> 15000]

But something goes wrong everytime I change slightly the fitting range (it finds a negative peak on the left side). Also the fit is not amazing. Can you suggest something to improve it? I also have to apply the same fit to similar dataset (where only the peak shifts) but again even if they are similar I get very different result. I would like to know how to make the fit more consistent and robust

;

(* Create background, arbitrary removal of peak *)
bg = Interpolation[Select[data, Not[2.3 < #[[1]] < 2.6] &], 
   InterpolationOrder -> 2];
(* Create background correction function *)
bgcorr[pt_] := {First@pt, Last@pt - bg[First@pt]}
(* Create background corrected dataset *)
corr = bgcorr /@ data;
(* Fit your corrected data with a model *)
nlm = NonlinearModelFit[corr, 
   a PDF[SkewNormalDistribution[b, c, d], x], {{a, 0.35}, {b, 
     2.5}, {c, 0.2}, {d, 0.5}}, x];
(* Show off the results *)
Plot[nlm[x] + bg[x], {x, 2.2, 3.4}, Epilog -> Point /@ data, 
 PlotLabel -> nlm["BestFitParameters"]]