Scrapy manually sending item to pipeline

Sometime, we need send result item to pipeline manually ex, entering input data to the database. This article will detail the step to do it

Step 1: Create a function to get pipeline instance

def get_pipeline_instance(spider, pipeline_class):
    manager = spider.crawler.engine.scraper.itemproc
    for pipe in manager.middlewares:
        if isinstance(pipe, pipeline_class):
            return pipe
        else:
            return None

def get_pipeline_instance(spider, pipeline_class):

manager = spider.crawler.engine.scraper.itemproc

for pipe in manager.middlewares:

if isinstance(pipe, pipeline_class):

return pipe

else:

return None

Step 2: Call the pipeline to send an item

try:
    # Manually call the filter
    f = get_pipeline_instance(self, YourPineLineClass)
    f.process_item(data_item, self)
except:
    pass

try:

# Manually call the filter

f = get_pipeline_instance(self, YourPineLineClass)

f.process_item(data_item, self)

except:

pass

Hope it help. Thanks!!!

etuannv

I am a Freelancer in programming specifically Python Scripting, Web scraping, and Web automation with 10+ years of experience.

Scrapy manually sending item to pipeline

Step 1: Create a function to get pipeline instance

Step 2: Call the pipeline to send an item

Other articles

Leave a Reply Cancel reply