Python Scrapy how to save data in different files

Saving your items into a file named after the page you found them in is (afaik) not supported in settings. If you wanted to achieve this, you could create your own functionality for that with python's open function and csv.writer in your parse method. An alternate option would be to write an item pipeline which manages different item exporters for different files.

What you can do with settings however is limit the number of items in a file with the FEED_EXPORT_BATCH_ITEM_COUNT setting, which is supported since of Scrapy version 2.3.
Overwriting instead of appending to a file can also be done since of Scrapy 2.4. In FEEDS you can set overwrite to True as demonstrated shortly.

If you were to replace your custom_settings with the following, it would produce files with 10 items each named page- followed by the batch_id, which starts with one. So your first 3 files would be named page-1.csv, page-2.csv and page-3.csv.

    custom_settings = {
        'FEED_EXPORT_BATCH_ITEM_COUNT': 10,
        'FEEDS' : {
            'page-%(batch_id)d.csv' : {
                'format' : 'csv',
                'store_empty' : True,
                'overwrite': True
            }
        }
    }

---

Implementing as pipeline

If you wanted to implement this using an item pipeline, you could save the page number you are on in the dictionary you return, which then gets processed and removed by the item pipeline.

The pipeline in your pipelines.py (based on this example) could then look like this:

from scrapy.exporters import CsvItemExporter


class PerFilenameExportPipeline:
    """Distribute items across multiple CSV files according to their 'page' field"""

    def open_spider(self, spider):
        self.filename_to_exporter = {}

    def close_spider(self, spider):
        for exporter in self.filename_to_exporter.values():
            exporter.finish_exporting()

    def _exporter_for_item(self, item):
        filename = 'page-' + str(item['page_no'])
        del item['page_no']
        if filename not in self.filename_to_exporter:
            f = open(f'{filename}.csv', 'wb')
            exporter = CsvItemExporter(f)
            exporter.start_exporting()
            self.filename_to_exporter[filename] = exporter
        return self.filename_to_exporter[filename]

    def process_item(self, item, spider):
        exporter = self._exporter_for_item(item)
        exporter.export_item(item)
        return item

To your spider you would then need to add a routine to get the page you are on as well as setting the pipeline in your custom_settings, which you could do like the following:

import scrapy
from ..pipelines import PerFilenameExportPipeline


class spidey(scrapy.Spider):
    name = "idk"
    custom_settings = {
        'ITEM_PIPELINES': {
            PerFilenameExportPipeline: 100
        }
    }
    
    def start_requests(self):
        yield scrapy.Request("http://quotes.toscrape.com/", cb_kwargs={'page_no': 1})

    def parse(self, response, page_no):
        for qts in response.xpath("//*[@class="quote"]"):
            yield {
                'page_no': page_no,
                'author' : qts.xpath("./span[2]/small/text()").get(),
                'quote' : qts.xpath("./*[@class="text"]/text()").get()
            }

        next_pg = response.xpath('//li[@class="next"]/a/@href').get()      
        if next_pg is not None:
            yield response.follow(next_pg, cb_kwargs={'page_no': page_no + 1})

However, there is one issue with this. The last file (page-10.csv) stays empty for reasons beyond my comprehension. I have asked why that could be here.