Tell me more ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

i am writing a scraper, using scrapy , in which i need to populate item with default values at first , is there any good way to populate items with default values other then my code logic

 def sp_offer_page(self, response):
            item = MallCrawlerItem()
            for key, value in self.get_default_item_dict().iteritems():
                item[key] = value
            item['is_coupon'] = 1
            item['mall'] = 'abc'
            # some other logic 
            yield item

    def get_default_item_dict(self):     # populate item with Default Values
        return {'mall': 'null', 'store': 'null', 'bonus': 'null',
                'per_action': 0, 'more_than': 0,
                'up_to': 0, 'deal_url': 'null', 'category': 'null', 'expiration': 'null', 'discount': 'null',
                'is_coupon': 0, 'code': 'null'}
share|improve this question

2 Answers

 def sp_offer_page(self, response):
            item = MallCrawlerItem()

If MallCrawlterItem is your class, it might make more sense to have it set everything to defaults.

            for key, value in self.get_default_item_dict().iteritems():
                item[key] = value

MallCrawlerItem appears to be dict-like. dict's have a method update, which can replace this loop. If MallCrawlerItem also has this method, you can use it.

            item['is_coupon'] = 1
            item['mall'] = 'abc'

Why not part of the defaults?

            # some other logic 
            yield item

This doesn't make sense here. I guess this function has been abbreviated.

    def get_default_item_dict(self):     # populate item with Default Values
        return {'mall': 'null', 'store': 'null', 'bonus': 'null',
                'per_action': 0, 'more_than': 0,
                'up_to': 0, 'deal_url': 'null', 'category': 'null', 'expiration': 'null', 'discount': 'null',
                'is_coupon': 0, 'code': 'null'}

I'd put this dictionary in a global constant, not put it inline with your code. I'd also put a line after all the commas. I think that'll be easier to read.

share|improve this answer

Item has default values. Why don't you use them?

share|improve this answer
i tried default values in item but its not working like category = Field(default='null') – akhter wahab Apr 3 '12 at 7:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.