Folders (Repo)
Specification Tab: S3 Folders (Repo)
- List of S3 Keys: A comma delimited keys of s3 keys (folders) to crawl.
- Bucket Name: The bucket where the keys are located
- Retrieve File Tags: File tags will be added as metadata with prefix “tag.”
Basic Configuration (Output)
Specification Tab: S3 Basic Configuration (Output)
Note: There are no actual folders in S3. All files in S3 have a “key”,
which includes their entire path. The folder path and bucket properties simply prepend
these values to each files’ keys
- Output Folder Path: Output folder key. Will be prepended to all document parent paths to make keys.
- Bucket Name: The bucket name that will be prepended to all keys.
- Includes Unmapped Properties: Will apply all metadata on the document without mapping
- Use GZip: Sets whether gzip decompression should be used when receiving HTTP responses.
- Do not generate XML when Outputting to S3: Like the BFS Connector, the S3 Connector outputs metadata as separate files in the form of [filename].metadata.properties.xml. Check this box if you wish for it to only output files.
- Use Transfer Manager: If migrating larger files, the S3 APIs offer a transfer manager to ensure more stable uploads
- Stage Binary to Filesystem: To avoid issues with disconnects from the source, this will temporarily store file content in the Tomcat temp folder before uploading it.
- Date/DateTime Format: How to format the mapped fields of this type before upload.
Note: If migrating large files to S3 it is
recommended that you check Use Transfer Manager AND Stage Binary to
Filesystem. If you use the Transfer Manager without staging the file,
all file uploads will be single threaded by the Transfer Manager.
Advanced Configuration (Output)
Specification Tab: S3 Advanced Configuration (Output)
- Max Connections: The maximum number of connections the client can open. Adjusting this can cause changes in performance
- Multi-value Separator: Some documents have fields that contain multiple values.S3 does not support this, and will use this separator to form a list of these values as a string before upload.
- Encrypt Object Server Side: Will encrypt uploaded files using AES 256 Encryption
- Disable Chunked Encoding: Will remove the transfer-encoding:chunked header from all requests
- Set Path Style Access: Refer to Amazon’s page for more information on this option
- Object Metadata Fields: A Comma delimited list of fields to add to the S3 Object as User Metadata.