Problems with deleting file from sftp folder using Python

Question

I am trying to move files into a backup folder which is on the same directory level as the folder where the files are stored. The downloading of the file into a local directory and uploading the file back to the backup folder works but the files can't be deleted from the existing remote folder. I wish to remove the files one by one after each file has been processed instead of storing the paths in a list and removing all at once. I have no idea what is wrong. Hope to seek advice.

Error reading file test_abs_2022135201552.pslog: Bad message

Below is a snippet of the code. I have tested that the 'put' and 'get' part of the code works. The line sftp.remove(file_path) is the one giving problems.

source_path = "/project/"
local_path = r'C:\Users\Desktop\project'

 df_staging = pd.DataFrame()
    
def main():


    transport = paramiko.Transport((sftp_host, sftp_port))
    transport.connect(username=sftp_username, password=sftp_password)
    sftp = paramiko.SFTPClient.from_transport(transport)

    global df_staging

    try:

        main_folders = main_folders = sftp.listdir(source_path) 

        for main_folder in main_folders:
            main_folder = os.path.join(source_path, main_folder) 
            station_folders = sftp.listdir(main_folder_path) # List the station folder names 

            # To create a backup folder if it does not exist in remote server
            if 'backup' not in sftp.listdir(main_folder_path):
                sftp.mkdir(os.path.join(main_folder_path, 'backup'))
                print("Backup folder created")
                
            # To create family folder if it does not exist in local dir
            if main_folder not in os.listdir(local_path):
                os.mkdir(os.path.join(local_path, main_folder))

            for station_folder in station_folders:
                if station_folder != 'backup':
                    station_folder_path = os.path.join(main_folder_path, station_folder) 
                    remote_files = sftp.listdir(station_folder_path) 


                   local_file_path = os.path.join(local_path, main_folder, file)
                   remote_backup_path = os.path.join(main_folder_path, 'backup', file)
                   # Transfer file between remote to local
                   sftp.get(file_path, local_file_path)
                   # Transfer file back from local to remote
                   sftp.put(local_file_path, remote_backup_path)
                   # Remove file from remote station folder (This line is giving me the errors!!!)
                   sftp.remove(file_path)
                                            
                   print(f"Done transfering: {file} ")
                   print("\n")
                                        
                   except Exception as e:
                      print(f"Error reading file {file}: {str(e)}")


                    lst_files_to_delete = [path for path in lst_all_files if path not in lst_files_to_transfer]
                    print("Files to delete:", lst_files_to_delete)

                    for delete_file_path in lst_files_to_delete:
                        print(f"Deleting file: {delete_file_path}")
                        sftp.remove(delete_file_path)
                        print(f"File deleted successfully")

This line: # Remove file from remote station folder sftp.remove(file_path) — Jolin
– Jolin, Commented Sep 11, 2023 at 6:24
No idea. My guess is the code was written for another server or you need additional configuration. Does the documentation say anything? Can you check the library code? — Thorbjørn Ravn Andersen
– Thorbjørn Ravn Andersen, Commented Sep 11, 2023 at 7:47
Do you need the for loop to reproduce the problem? Do you need the list_dir, mkdir or put? If not, reduce your code to minimal reproducible example. We are not interested in your whole program. I assume just the get is useful here, as it shows that the file_path is valid. + Can you delete the file using sftp? If you can, show us equivalent transcript of the sftp session, with the same paths as your code is using. — Martin Prikryl
– Martin Prikryl, Commented Sep 11, 2023 at 8:21

Jolin · Accepted Answer · 2023-09-12 06:29:00Z

-1

As my file was still opened to be read in the main code, the program was unable to delete the file. The problem is resolved by moving the remove line out of a loop that is reading the file

answered Sep 12, 2023 at 6:29

Jolin

276 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ableitung Over a year ago

Always close your file handles after you're done using them to avoid such problems in the future.

Martin Prikryl Over a year ago

What "main code"?

Jolin · Accepted Answer · 2023-09-14 02:07:14Z

-1

Essentially, sftp.remove can only work when the file is closed. In the previous example where I encountered the problem, sftp.remove was placed inside with sftp.open. After moving it out, it works. Below is the portion of the main code that I was referring to:

with sftp.open(file_path, 'r') as remote_file:
    # Main Code


### After file is closed, perform file upload and removal only after uploaded to DB ###
local_file_path = os.path.join(local_path, main_folder, file)
remote_backup_path = os.path.join(main_folder_path, 'backup', file)
# Transfer file between remote to local
sftp.get(file_path, local_file_path)
# Transfer file back from local to remote
sftp.put(local_file_path, remote_backup_path)
# Remove file from remote station folder (***Only works when file is not open!)
sftp.remove(file_path)

answered Sep 14, 2023 at 2:07

Jolin

276 bronze badges

1 Comment

Martin Prikryl Over a year ago

Well, there's no sftp.open in your original code. You have failed to post minimal reproducible example.

Collectives™ on Stack Overflow

Problems with deleting file from sftp folder using Python

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related