parallel deletion (without following symlinks) using jwalk #43

Byron · 2020-03-15T23:29:11Z

Previously jwalk would follow symlinks, so we could not use it and instead implemented our own traversal.

Now with jwalk 0.5, this is supported and we can use our standard jwalker to do the job.

Lines 305 to 340 in 543f7f3

    
           let mut files_or_dirs = vec![path]; 
        
           let mut dirs = Vec::new(); 
        
           let mut num_errors = 0; 
        
           while let Some(path) = files_or_dirs.pop() { 
        
               let assume_symlink_to_try_deletion = true; 
        
               let is_symlink = path 
        
                   .symlink_metadata() 
        
                   .map(|m| m.file_type().is_symlink()) 
        
                   .unwrap_or(assume_symlink_to_try_deletion); 
        
               if is_symlink { 
        
                   // do not follow symlinks 
        
                   num_errors += into_error_count(fs::remove_file(&path)); 
        
                   continue; 
        
               } 
        
               match fs::read_dir(&path) { 
        
                   Ok(iterator) => { 
        
                       dirs.push(path); 
        
                       for entry in iterator { 
        
                           match entry.map_err(io_err_to_usize) { 
        
                               Ok(entry) => files_or_dirs.push(entry.path()), 
        
                               Err(c) => num_errors += c, 
        
                           } 
        
                       } 
        
                   } 
        
                   Err(ref e) if e.kind() == io::ErrorKind::Other => { 
        
                       // assume file, save IOps 
        
                       num_errors += into_error_count(fs::remove_file(path)); 
        
                       continue; 
        
                   } 
        
                   Err(_) => { 
        
                       num_errors += 1; 
        
                       continue; 
        
                   } 
        
               }; 
        
           }

The function above would have to change to use jwalk. It should be straightforward to obtain a walker with this function:

dua-cli/src/common.rs

Lines 162 to 184 in 0d6116e

    
           impl WalkOptions { 
        
               pub(crate) fn iter_from_path(&self, path: &Path) -> WalkDir { 
        
                   WalkDir::new(path) 
        
                       .follow_links(false) 
        
                       .sort(match self.sorting { 
        
                           TraversalSorting::None => false, 
        
                           TraversalSorting::AlphabeticalByFileName => true, 
        
                       }) 
        
                       .skip_hidden(false) 
        
                       .process_read_dir(|_, dir_entry_results| { 
        
                           dir_entry_results.iter_mut().for_each(|dir_entry_result| { 
        
                               if let Ok(dir_entry) = dir_entry_result { 
        
                                   dir_entry.client_state = Some(dir_entry.metadata()); 
        
                               } 
        
                           }) 
        
                       }) 
        
                       .parallelism(if self.threads == 0 { 
        
                           jwalk::Parallelism::RayonDefaultPool 
        
                       } else { 
        
                           jwalk::Parallelism::RayonNewPool(self.threads) 
        
                       }) 
        
               } 
        
           }

Byron added help wanted good first issue labels Mar 15, 2020

Byron added the enhancement label Jul 6, 2020

Byron / dua-cli

parallel deletion (without following symlinks) using jwalk #43

parallel deletion (without following symlinks) using jwalk #43

Byron commented Mar 15, 2020

Byron / dua-cli

Join GitHub today

parallel deletion (without following symlinks) using jwalk #43

parallel deletion (without following symlinks) using jwalk #43

Comments

Byron commented Mar 15, 2020

Essential cookies

Always active

Analytics cookies