The SharePoint library and document set only display different pages, but they both use the default view - All Documents. So, when you set a filter in the document set, you are actually setting a filter in the All-Documents view in the SharePoint library.
Navigating back using the browser's back button will not save your filter settings. Using the breadcrumb navigation in the library will retain your filter settings.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.