You can remove a query set by performing its removal in the Admin Console. There is no need to perform a refresh after the query set removal, however the JSON config file should be manually removed from the config directory.
During the refresh the JSON config files will be compared against the internal registry of query sets. If a query set in the registry does not have a corresponding JSON config file with the same tableName then a warning will be logged. The denormalized database table will NOT be dropped.