问题现象

使用S3存储作为Harbor存储时，上传单层layer较大镜像时，出现失败重试的现象

docker push harbordomain/library/centos
Using default tag: latest
The push refers to repository [harbordomain/library/centos]
9e607bb861a7: Retrying in 4 seconds 

相关信息

https://github.com/distribution/distribution/pull/3893
https://github.com/goharbor/harbor/pull/18769
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
https://docs.docker.com/registry/storage-drivers/s3/

故障原因

Multipart upload listings
You can list the parts of a specific multipart upload or all in-progress multipart uploads. The list parts operation returns the parts information that you have uploaded for a specific multipart upload. For each list parts request, Amazon S3 returns the parts information for the specified multipart upload, up to a maximum of 1,000 parts. If there are more than 1,000 parts in the multipart upload, you must send a series of list part requests to retrieve all the parts. Note that the returned list of parts doesn't include parts that haven't finished uploading. Using the list multipart uploads operation, you can obtain a list of multipart uploads that are in progress.
An in-progress multipart upload is an upload that you have initiated, but have not yet completed or stopped. Each request returns at most 1,000 multipart uploads. If there are more than 1,000 multipart uploads in progress, you must send additional requests to retrieve the remaining multipart uploads. Use the returned listing only for verification. Do not use the result of this listing when sending a complete multipart upload request. Instead, maintain your own list of the part numbers that you specified when uploading parts and the corresponding ETag values that Amazon S3 returns.

问题源于distribution，Harbor使用distribution作为registry；distribution使用S3 API的multipartupload方法上传镜像到存储，multipartupload一个请求最多支持1000个分片上传，超过1000个分片需要发送额外的请求处理，而distribution v2.8.1之前的版本缺少了对超过1000个分片的处理。

Harbor registry configmap中的chunksize参数（也是distribution的参数）决定了multipartupload的分片大小，缺省为1010241024=10485760=10M，所以单层laryer最大为10M*1000=10G。

解决方法

方法1

升级distribution到v2.8.2以上

方法2

调大registry configmap中的chunksize参数