Best practice for plots (pgfplots, gnuplot, etc.)

I had a similar problem. What I did is "externalise" the generation of the plots. You do this by including

usepgfplotslibrary{external}
tikzexternalize

In your preamble. You have to call Latex with the additional options: -shell-escape. This will cause the compiler to start from scratch for each tikz-picture so your compilation will no longer accumulate memory consumption. There will be an individual file for each picture and they will be nicely combined into one document as well.

This being said, 1M points is still too much. And frankly, I don't think it is necessary to have a nice quality plot. You should reduce that anyway. Play around with the number of points (You have it in Python anyway so it should not be that hard.) I try to find the limit by trial and error. You might want to create and debug the picture in a separate document in order to get the error-log.

I went through this problem years ago. In my cases I wanted to make sure I my tikz/pgfplots

1. could be done using axes as well as windows
2. were capable of showing a graph paper background
3. were able to handle multiple graphs at the same time
4. Could handle more complicated functions

Now simple functions, such as what you'd see in a high school are easily done with template 1: I posted Tikz plot x^a*cos(ln(x)). You'll see there are many settings where the axis environment is set. The line commented out %grid = both,%grid for major ticks (every integer) and minor ticks is preventing graph paper background. After experimenting a bit I found that the sagetex package was capable of the issue you are concerned with: handling more complicated functions. The documentation is found on CTAN here; the package gives you access to a computer algebra system (CAS), called SAGE, along with Python programming. From the CAS you get more complicated functions such as erf, riemann zeta, fourier, and so on. The SAGE documentation on built in functions is here. You also get accurate calculations, the ability to compute derivatives and integrals, and the ability to choose the points you graph. Since SAGE is not part of your LaTeX distribution it needs to be installed on your computer or you need to open a free Cocalc account. The free account is the easiest way to get up and running.

After pushing the Build button in Cocalc there is a 3 step process compilation process that goes on: first LaTeX is run. Second, SAGE runs on the content it needs to. Third, LaTeX is run on the final product consisting of LaTeX plus SAGE content. In order to get through compilation without errors I've found it easiest to create a raw string in Python which will take any calculations needed (such as coordinates in a plot) and create the text that will run in LaTeX. Template 2: for 2d plotting requiring SAGE is used in many posts on this site; for example: how to plot reciprocal of Gamma function in Latex. Notice that I imported the function with from sage.functions.other import gamma1 and used that in two ways. It's first plotted as a sageplot which is showing you the plot created by SAGE. The second plot shows the result when the coordinates are pushed through pgfplots to get the second plot shown. It's similar to the first template except now I'm putting the code through a string so SAGE can do the typesetting. Since my axis environment says output += r"grid = none," there will be no graph paper. In the post Plotting implicitly in Tikz you'll see the same basic template used to plot implicitly. Note the graph paper background comes from output += r"grid = both,". In addition to these 2 templates for plotting 2d, there is template 3: for 3d plots requiring SAGE in Tikz plotting 3d. Once you're comfortable with the process you can easily use SAGE as the engine for other plots not using the templates discussed. For example the post

1. Drawing Fourier expansion using pgfplots uses SAGE knowledge of Fourier series plus a different plot setup.
2. How to plot Lebesgue singular function in Tikz? uses Python programming more in creating the function.
3. Plotting function ℝ² →ℝ with pole at (0,0) smoothly shows how SAGE can plot using the specific points you want.
4. Drawing the dragon curve mentions how to increase your buffer: You'd change the compilation to something like buf_size=1000000 pdflatex -synctex=1 -interact=nonstopmode MyFile.tex.
5. Contour plot of arbitrary R^3 function without gnuplot shows how you can embed matplotlib plots in your document.
6. Given a list of primes, how can I plot π(x)? uses the built in prime function to plot a sequence.
7. Riemann Integral on Tikz with commands shows how to create a Python function which will generate left and right Riemann integrals.
8. How to automatically ask tikz to produce the coordinates and plot the graph from adjacency matrix creates vertices and edges type graph from an adjacency matrix.
9. Obtain a table of non-uniformly-distributed random 0 and 1 draws illustrates how a numpy array could processed to create an array of randomly created ones and zeros.
10. Projective Plot of a elliptic curve showing off SAGE 3d graphics.
11. How to plot this function containing ceiling in TikZ? shows how to plot a function with numerous dicontinuities.

In summary, you can use the sagetex package along with the three templates to create 2d/3d plots with the same look and feel, regardless of how complicated there are. The use of Python keeps code simpler to read and revise and even lets you force the plotting of specific points. You can easily increase the size of the buffer to handle more data. SAGE can handle differentiation as well as integration. You can also use matplotlib and SAGE plots if you want. There are other examples posted on this site you can search for as well.

What's best practice? I don't know that has an answer to satisfy everyone. However, the sagetex package can handle the problems you have mentioned.