(PC-39018)[API] refactor: optimize query for `get_expired_offers()` #20437

tconte-pass · 2025-11-28T17:25:47Z

🎯 Related Ticket or 🔧 Changes Made

La query telle qu'elle était avait un temps moyen d'exécution de ~2h (par batch de 10 000)
La nouvelle query s'est exécutée en :

~26s (sans offset, avec limit 10000)
~23s (avec offset 50000, limit 10000)

query plan : sans offset, avec limit 10000

Limit  (cost=47302.30..47823.14 rows=4378 width=8) (actual time=26068.276..26078.785 rows=10000 loops=1)                                                                                                                                                                                                                                                                                                                                                                                                             
   Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
   Buffers: shared hit=466945 read=115474 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
   I/O Timings: shared read=75069.028                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
   ->  Unique  (cost=47302.30..47823.14 rows=4378 width=8) (actual time=26068.274..26077.749 rows=10000 loops=1)                                                                                                                                                                                                                                                                                                                                                                                                      
         Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
         Buffers: shared hit=466945 read=115474 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                                           
         I/O Timings: shared read=75069.028                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
         ->  Gather Merge  (cost=47302.30..47812.19 rows=4378 width=8) (actual time=26068.273..26075.778 rows=10008 loops=1)                                                                                                                                                                                                                                                                                                                                                                                          
               Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
               Workers Planned: 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
               Workers Launched: 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
               Buffers: shared hit=466945 read=115474 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                                     
               I/O Timings: shared read=75069.028                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
               ->  Sort  (cost=46302.28..46306.84 rows=1824 width=8) (actual time=25957.319..25957.689 rows=4390 loops=3)                                                                                                                                                                                                                                                                                                                                                                                             
                     Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                     Sort Key: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                     Sort Method: quicksort  Memory: 385kB                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                     Buffers: shared hit=466945 read=115474 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                               
                     I/O Timings: shared read=75069.028                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                     Worker 0:  actual time=25902.180..25902.685 rows=5119 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                                    
                       Sort Method: quicksort  Memory: 385kB                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                       Buffers: shared hit=151681 read=37953 dirtied=123                                                                                                                                                                                                                                                                                                                                                                                                                                              
                       I/O Timings: shared read=24955.662                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                     Worker 1:  actual time=25902.378..25902.760 rows=5119 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                                    
                       Sort Method: quicksort  Memory: 385kB                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                       Buffers: shared hit=154726 read=38791 dirtied=105                                                                                                                                                                                                                                                                                                                                                                                                                                              
                       I/O Timings: shared read=24980.460                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                     ->  Nested Loop Anti Join  (cost=1.72..46203.48 rows=1824 width=8) (actual time=66.314..25943.799 rows=13521 loops=3)                                                                                                                                                                                                                                                                                                                                                                            
                           Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                           Buffers: shared hit=466927 read=115474 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                         
                           I/O Timings: shared read=75069.028                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                           Worker 0:  actual time=27.890..25889.446 rows=12580 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                                
                             Buffers: shared hit=151672 read=37953 dirtied=123                                                                                                                                                                                                                                                                                                                                                                                                                                        
                             I/O Timings: shared read=24955.662                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                           Worker 1:  actual time=116.270..25888.230 rows=14483 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                               
                             Buffers: shared hit=154717 read=38791 dirtied=105                                                                                                                                                                                                                                                                                                                                                                                                                                        
                             I/O Timings: shared read=24980.460                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                           ->  Nested Loop  (cost=1.15..43642.00 rows=1843 width=8) (actual time=8.944..20983.814 rows=16367 loops=3)                                                                                                                                                                                                                                                                                                                                                                                 
                                 Output: offer.id                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                 Inner Unique: true                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                 Buffers: shared hit=253491 read=93362 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                                    
                                 I/O Timings: shared read=61245.339                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                 Worker 0:  actual time=2.346..20854.785 rows=15676 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                           
                                   Buffers: shared hit=81778 read=30683 dirtied=123                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                   I/O Timings: shared read=20281.911                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                                 Worker 1:  actual time=18.060..20800.274 rows=16990 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                          
                                   Buffers: shared hit=83411 read=31057 dirtied=105                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                   I/O Timings: shared read=20236.637                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                                 ->  Parallel Index Scan using "ix_stock_bookingLimitDatetime_partial" on public.stock  (cost=0.57..38244.74 rows=2083 width=8) (actual time=8.597..8859.613 rows=18223 loops=3)                                                                                                                                                                                                                                                                                                      
                                       Output: stock."offerId"                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                                       Index Cond: ((stock."bookingLimitDatetime" >= ((now())::timestamp without time zone - '2 days'::interval)) AND (stock."bookingLimitDatetime" <= (now())::timestamp without time zone))                                                                                                                                                                                                                                                                                         
                                       Filter: (stock."isSoftDeleted" IS FALSE)                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                       Rows Removed by Filter: 634                                                                                                                                                                                                                                                                                                                                                                                                                                                    
                                       Buffers: shared hit=32332 read=41169 dirtied=365                                                                                                                                                                                                                                                                                                                                                                                                                               
                                       I/O Timings: shared read=25985.390                                                                                                                                                                                                                                                                                                                                                                                                                                             
                                       Worker 0:  actual time=2.223..8765.200 rows=17392 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                      
                                         Buffers: shared hit=12009 read=13491 dirtied=123                                                                                                                                                                                                                                                                                                                                                                                                                             
                                         I/O Timings: shared read=8563.663                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                       Worker 1:  actual time=17.936..8632.634 rows=18557 loops=1                                                                                                                                                                                                                                                                                                                                                                                                                     
                                         Buffers: shared hit=8214 read=13468 dirtied=105                                                                                                                                                                                                                                                                                                                                                                                                                              
                                         I/O Timings: shared read=8443.558                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                 ->  Index Scan using offer_pkey on public.offer  (cost=0.57..2.59 rows=1 width=8) (actual time=0.662..0.662 rows=1 loops=54670)                                                                                                                                                                                                                                                                                                                                                      
                                       Output: offer."dateModifiedAtLastProvider", offer.id, offer."dateCreated", offer."productId", offer."venueId", offer."lastProviderId", offer."bookingEmail", offer."isActive", offer.name, offer.description, offer.url, offer."durationMinutes", offer."isNational", offer."isDuo", offer."fieldsUpdated", offer."withdrawalDetails", offer."audioDisabilityCompliant", offer."mentalDisabilityCompliant", offer."motorDisabilityCompliant", offer."visualDisabilityCompli... 
                                       Index Cond: (offer.id = stock."offerId")                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                       Filter: ((offer."publicationDatetime" IS NOT NULL) AND (offer."publicationDatetime" <= now()))                                                                                                                                                                                                                                                                                                                                                                                 
                                       Rows Removed by Filter: 0                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                       Buffers: shared hit=221159 read=52193                                                                                                                                                                                                                                                                                                                                                                                                                                          
                                       I/O Timings: shared read=35259.949                                                                                                                                                                                                                                                                                                                                                                                                                                             
                                       Worker 0:  actual time=0.692..0.692 rows=1 loops=17392                                                                                                                                                                                                                                                                                                                                                                                                                         
                                         Buffers: shared hit=69769 read=17192                                                                                                                                                                                                                                                                                                                                                                                                                                         
                                         I/O Timings: shared read=11718.248                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                       Worker 1:  actual time=0.653..0.653 rows=1 loops=18557                                                                                                                                                                                                                                                                                                                                                                                                                         
                                         Buffers: shared hit=75197 read=17589                                                                                                                                                                                                                                                                                                                                                                                                                                         
                                         I/O Timings: shared read=11793.079                                                                                                                                                                                                                                                                                                                                                                                                                                           
                           ->  Index Only Scan using ix_stock_bookability on public.stock stock_1  (cost=0.57..7.04 rows=8 width=8) (actual time=0.299..0.299 rows=0 loops=49102)                                                                                                                                                                                                                                                                                                                                     
                                 Output: stock_1."offerId", stock_1."beginningDatetime", stock_1."isSoftDeleted", stock_1."bookingLimitDatetime", stock_1.quantity, stock_1."dnBookedQuantity"                                                                                                                                                                                                                                                                                                                        
                                 Index Cond: ((stock_1."offerId" = offer.id) AND (stock_1."isSoftDeleted" = false) AND (stock_1."bookingLimitDatetime" > (now())::timestamp without time zone))                                                                                                                                                                                                                                                                                                                       
                                 Heap Fetches: 6649                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                 Buffers: shared hit=213436 read=22112                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                 I/O Timings: shared read=13823.689                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                 Worker 0:  actual time=0.318..0.318 rows=0 loops=15676                                                                                                                                                                                                                                                                                                                                                                                                                               
                                   Buffers: shared hit=69894 read=7270                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                   I/O Timings: shared read=4673.751                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                 Worker 1:  actual time=0.296..0.296 rows=0 loops=16990                                                                                                                                                                                                                                                                                                                                                                                                                               
                                   Buffers: shared hit=71306 read=7734                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                   I/O Timings: shared read=4743.823                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 Query Identifier: -6169886911323517090                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
 Planning:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
   Buffers: shared hit=625                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
 Planning Time: 6.889 ms                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
 Execution Time: 26079.631 ms

query plan : avec offset 50000, avec limit 10000

Limit  (cost=47851.94..47852.05 rows=1 width=8) (actual time=23609.971..23611.174 rows=0 loops=1)                                                                                                 
   Output: offer.id                                                                                                                                                                                
   Buffers: shared hit=460042 read=113377 dirtied=17                                                                                                                                               
   I/O Timings: shared read=67806.788                                                                                                                                                              
   ->  Unique  (cost=47330.86..47851.94 rows=4380 width=8) (actual time=23585.581..23608.309 rows=40529 loops=1)                                                                                   
         Output: offer.id                                                                                                                                                                          
         Buffers: shared hit=460042 read=113377 dirtied=17                                                                                                                                         
         I/O Timings: shared read=67806.788                                                                                                                                                        
         ->  Gather Merge  (cost=47330.86..47840.99 rows=4380 width=8) (actual time=23585.578..23597.328 rows=40704 loops=1)                                                                       
               Output: offer.id                                                                                                                                                                    
               Workers Planned: 2                                                                                                                                                                  
               Workers Launched: 2                                                                                                                                                                 
               Buffers: shared hit=460042 read=113377 dirtied=17                                                                                                                                   
               I/O Timings: shared read=67806.788                                                                                                                                                  
               ->  Sort  (cost=46330.84..46335.40 rows=1825 width=8) (actual time=23536.759..23538.769 rows=13568 loops=3)                                                                         
                     Output: offer.id                                                                                                                                                              
                     Sort Key: offer.id                                                                                                                                                            
                     Sort Method: quicksort  Memory: 385kB                                                                                                                                         
                     Buffers: shared hit=460042 read=113377 dirtied=17                                                                                                                             
                     I/O Timings: shared read=67806.788                                                                                                                                            
                     Worker 0:  actual time=23523.386..23526.735 rows=12991 loops=1                                                                                                                
                       Sort Method: quicksort  Memory: 385kB                                                                                                                                       
                       Buffers: shared hit=144109 read=37604 dirtied=4                                                                                                                             
                       I/O Timings: shared read=22602.981                                                                                                                                          
                     Worker 1:  actual time=23502.099..23503.341 rows=14013 loops=1                                                                                                                
                       Sort Method: quicksort  Memory: 385kB                                                                                                                                       
                       Buffers: shared hit=158732 read=37658 dirtied=5                                                                                                                             
                       I/O Timings: shared read=22546.102                                                                                                                                          
                     ->  Nested Loop Anti Join  (cost=1.72..46231.98 rows=1825 width=8) (actual time=13.977..23523.771 rows=13568 loops=3)                                                         
                           Output: offer.id                                                                                                                                                        
                           Buffers: shared hit=460024 read=113377 dirtied=17                                                                                                                       
                           I/O Timings: shared read=67806.788                                                                                                                                      
                           Worker 0:  actual time=10.958..23510.888 rows=12991 loops=1                                                                                                             
                             Buffers: shared hit=144100 read=37604 dirtied=4                                                                                                                       
                             I/O Timings: shared read=22602.981                                                                                                                                    
                           Worker 1:  actual time=6.965..23488.002 rows=14013 loops=1                                                                                                              
                             Buffers: shared hit=158723 read=37658 dirtied=5                                                                                                                       
                             I/O Timings: shared read=22546.102                                                                                                                                    
                           ->  Nested Loop  (cost=1.15..43669.11 rows=1844 width=8) (actual time=8.408..18897.967 rows=16406 loops=3)                                                              
                                 Output: offer.id                                                                                                                                                  
                                 Inner Unique: true                                                                                                                                                
                                 Buffers: shared hit=246336 read=91203 dirtied=17                                                                                                                  
                                 I/O Timings: shared read=54986.656                                                                                                                                
                                 Worker 0:  actual time=7.963..18906.720 rows=15415 loops=1                                                                                                        
                                   Buffers: shared hit=77491 read=30316 dirtied=4                                                                                                                  
                                   I/O Timings: shared read=18341.759                                                                                                                              
                                 Worker 1:  actual time=5.563..18709.842 rows=17119 loops=1                                                                                                        
                                   Buffers: shared hit=83862 read=29918 dirtied=5                                                                                                                  
                                   I/O Timings: shared read=18137.527                                                                                                                              
                                 ->  Parallel Index Scan using "ix_stock_bookingLimitDatetime_partial" on public.stock  (cost=0.57..38266.67 rows=2085 width=8) (actual time=6.631..7967.317 rows=1
                                       Output: stock."offerId"                                                                                                                                     
                                       Index Cond: ((stock."bookingLimitDatetime" >= ((now())::timestamp without time zone - '2 days'::interval)) AND (stock."bookingLimitDatetime" <= (now())::tim
                                       Filter: (stock."isSoftDeleted" IS FALSE)                                                                                                                    
                                       Rows Removed by Filter: 635                                                                                                                                 
                                       Buffers: shared hit=23220 read=40307 dirtied=17                                                                                                             
                                       I/O Timings: shared read=23335.200                                                                                                                          
                                       Worker 0:  actual time=5.489..8043.305 rows=17311 loops=1                                                                                                   
                                         Buffers: shared hit=7659 read=13592 dirtied=4                                                                                                             
                                         I/O Timings: shared read=7848.532                                                                                                                         
                                       Worker 1:  actual time=4.040..7529.311 rows=18622 loops=1                                                                                                   
                                         Buffers: shared hit=8016 read=12653 dirtied=5                                                                                                             
                                         I/O Timings: shared read=7346.406                                                                                                                         
                                 ->  Index Scan using offer_pkey on public.offer  (cost=0.57..2.59 rows=1 width=8) (actual time=0.596..0.596 rows=1 loops=54802)                                   
                                       Output: offer."dateModifiedAtLastProvider", offer.id, offer."dateCreated", offer."productId", offer."venueId", offer."lastProviderId", offer."bookingEmail",
                                       Index Cond: (offer.id = stock."offerId")                                                                                                                    
                                       Filter: ((offer."publicationDatetime" IS NOT NULL) AND (offer."publicationDatetime" <= now()))                                                              
                                       Rows Removed by Filter: 0                                                                                                                                   
                                       Buffers: shared hit=223116 read=50896                                                                                                                       
                                       I/O Timings: shared read=31651.456                                                                                                                          
                                       Worker 0:  actual time=0.625..0.625 rows=1 loops=17311                                                                                                      
                                         Buffers: shared hit=69832 read=16724                                                                                                                      
                                         I/O Timings: shared read=10493.227                                                                                                                        
                                       Worker 1:  actual time=0.598..0.598 rows=1 loops=18622                                                                                                      
                                         Buffers: shared hit=75846 read=17265                                                                                                                      
                                         I/O Timings: shared read=10791.121                                                                                                                        
                           ->  Index Only Scan using ix_stock_bookability on public.stock stock_1  (cost=0.57..7.04 rows=8 width=8) (actual time=0.279..0.279 rows=0 loops=49217)                  
                                 Output: stock_1."offerId", stock_1."beginningDatetime", stock_1."isSoftDeleted", stock_1."bookingLimitDatetime", stock_1.quantity, stock_1."dnBookedQuantity"     
                                 Index Cond: ((stock_1."offerId" = offer.id) AND (stock_1."isSoftDeleted" = false) AND (stock_1."bookingLimitDatetime" > (now())::timestamp without time zone))    
                                 Heap Fetches: 6651                                                                                                                                                
                                 Buffers: shared hit=213688 read=22174                                                                                                                             
                                 I/O Timings: shared read=12820.132                                                                                                                                
                                 Worker 0:  actual time=0.295..0.295 rows=0 loops=15415                                                                                                            
                                   Buffers: shared hit=66609 read=7288                                                                                                                             
                                   I/O Timings: shared read=4261.222                                                                                                                               
                                 Worker 1:  actual time=0.276..0.276 rows=0 loops=17119                                                                                                            
                                   Buffers: shared hit=74861 read=7740                                                                                                                             
                                   I/O Timings: shared read=4408.575                                                                                                                               
 Query Identifier: -6835815546325747147                                                                                                                                                            
 Planning:                                                                                                                                                                                         
   Buffers: shared hit=625                                                                                                                                                                         
 Planning Time: 6.673 ms                                                                                                                                                                           
 Execution Time: 23611.438 ms

github-actions · 2025-11-28T17:28:14Z

mypy cop report: 174 (master) ↘ 173 (your branch)

Have yourself a merry little break and learn a few facts about the year 173.

api/src/pcapi/core/offers/repository.py

tconte-pass force-pushed the pc-39018-optimize-get-expired-offers-query branch 2 times, most recently from 6cfd448 to a096d49 Compare December 1, 2025 07:03

jcicurel-pass reviewed Dec 1, 2025

View reviewed changes

api/src/pcapi/core/offers/repository.py Outdated Show resolved Hide resolved

api/src/pcapi/core/offers/repository.py Outdated Show resolved Hide resolved

prouzet-pass reviewed Dec 1, 2025

View reviewed changes

api/src/pcapi/core/offers/repository.py Outdated Show resolved Hide resolved

api/src/pcapi/core/offers/repository.py Outdated Show resolved Hide resolved

(PC-39018)[API] refactor: optimize query for get_expired_offers()

a8491d9

tconte-pass force-pushed the pc-39018-optimize-get-expired-offers-query branch from a096d49 to a8491d9 Compare December 1, 2025 09:32

prouzet-pass approved these changes Dec 1, 2025

View reviewed changes

jcicurel-pass approved these changes Dec 1, 2025

View reviewed changes

tconte-pass merged commit 417159c into master Dec 1, 2025
32 checks passed

tconte-pass deleted the pc-39018-optimize-get-expired-offers-query branch December 1, 2025 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(PC-39018)[API] refactor: optimize query for `get_expired_offers()` #20437

(PC-39018)[API] refactor: optimize query for `get_expired_offers()` #20437

Uh oh!

tconte-pass commented Nov 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

(PC-39018)[API] refactor: optimize query for get_expired_offers() #20437

(PC-39018)[API] refactor: optimize query for get_expired_offers() #20437

Uh oh!

Conversation

tconte-pass commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 Related Ticket or 🔧 Changes Made

Uh oh!

github-actions bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

(PC-39018)[API] refactor: optimize query for `get_expired_offers()` #20437

(PC-39018)[API] refactor: optimize query for `get_expired_offers()` #20437

tconte-pass commented Nov 28, 2025 •

edited

Loading

github-actions bot commented Nov 28, 2025 •

edited

Loading