Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dry run output is missing information in 10.27.1 version. Earlier it used to work until 10.26 version #2918

Open
rnandurimsft opened this issue Jan 21, 2025 · 2 comments
Assignees
Milestone

Comments

@rnandurimsft
Copy link

rnandurimsft commented Jan 21, 2025

Copy command -

copy "https://.dfs.core.windows.net/azcopy-unit-tests/source-new/*?" "https://.dfs.core.windows.net/azcopy-unit-tests/sink/AzCopy_Gen2Source_WithAccountKey_Gen2Sink_WithAccountKey_1/?" --dry-run --check-length=false --s2s-preserve-access-tier=false --output-type=json --recursive=true

The output used to be like below in 10.26.0

{"TimeStamp":"2025-01-21T12:37:15.1896011+05:30","MessageType":"Info","MessageContent":"INFO: Scanning...","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}
{"TimeStamp":"2025-01-21T12:37:18.317308+05:30","MessageType":"Dryrun","MessageContent":"{"Source":"/folder1","Destination":"/folder1","EntityType":1,"LastModifiedTime":"2024-07-19T05:20:09Z","SourceSize":0,"ContentType":"","ContentEncoding":"","ContentDisposition":"","ContentLanguage":"","CacheControl":"","ContentMD5":null,"Metadata":{"Hdi_isfolder":"true"},"BlobType":"BlockBlob","BlobTier":"","BlobVersionID":"","BlobTags":null,"BlobSnapshotID":""}","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}
{"TimeStamp":"2025-01-21T12:37:18.3178145+05:30","MessageType":"Dryrun","MessageContent":"{"Source":"/dataset.csv","Destination":"/dataset.csv","EntityType":0,"LastModifiedTime":"2024-07-19T05:20:22Z","SourceSize":2727,"ContentType":"application/octet-stream","ContentEncoding":"","ContentDisposition":"","ContentLanguage":"","CacheControl":"","ContentMD5":null,"Metadata":null,"BlobType":"BlockBlob","BlobTier":"","BlobVersionID":"","BlobTags":null,"BlobSnapshotID":""}","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}

The output now with 10.27.1 looks like below

{"TimeStamp":"2025-01-21T12:33:09.8304079+05:30","MessageType":"Info","MessageContent":"INFO: Scanning...","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}
{"TimeStamp":"2025-01-21T12:33:12.9085524+05:30","MessageType":"Dryrun","MessageContent":"{"EntityType":"Folder","BlobType":"BlockBlob","FromTo":"BlobFSBlobFS","Source":"https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/source-new/folder1\",\"Destination\":\"https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/sink/AzCopy_Gen2Source_WithAccountKey_Gen2Sink_WithAccountKey_1/folder1\"}","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}
{"TimeStamp":"2025-01-21T12:33:12.9085524+05:30","MessageType":"Dryrun","MessageContent":"{"EntityType":"File","BlobType":"BlockBlob","FromTo":"BlobFSBlobFS","Source":"https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/source-new/dataset.csv\",\"Destination\":\"https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/sink/AzCopy_Gen2Source_WithAccountKey_Gen2Sink_WithAccountKey_1/dataset.csv\"}","PromptDetails":{"PromptType":"","ResponseOptions":null,"PromptTarget":""}}

MessageContent contract is missing fields like SourceSize

If we just diff MessageContent outputs, below is the output for clarity

10.26 version

{
"Source": "/folder1/folder2/folder3/simpledataset.txt",
"Destination": "/folder1/folder2/folder3/simpledataset.txt",
"EntityType": 0,
"LastModifiedTime": "2024-07-19T05:20:19Z",
"SourceSize": 42,
"ContentType": "application/octet-stream",
"ContentEncoding": "",
"ContentDisposition": "",
"ContentLanguage": "",
"CacheControl": "",
"ContentMD5": null,
"Metadata": null,
"BlobType": "BlockBlob",
"BlobTier": "",
"BlobVersionID": "",
"BlobTags": null,
"BlobSnapshotID": ""
}

10.27.1

{
"EntityType": "File",
"BlobType": "BlockBlob",
"FromTo": "BlobFSBlobFS",
"Source": "https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/source-new/dataset.csv",
"Destination": "https://REDACTED.dfs.core.windows.net/azcopy-unit-tests/sink/AzCopy_Gen2Source_WithAccountKey_Gen2Sink_WithAccountKey_1/dataset.csv"
}

We see many fields missing here. Could you please include all the missing fields.

Logs

10.27.1.log
10.27.1-output.txt
10.27.1-scanning.log

10.26-output.txt
10.26-scanning.log
10.26.log

@dphulkar-msft
Copy link
Collaborator

Hello @rnandurimsft,
As a remediation measure, azcopy team will restore the missing outputs in a PR scheduled for the next release i,e. 10.29.0

@dphulkar-msft dphulkar-msft added this to the 10.29.0 milestone Jan 22, 2025
@rnandurimsft
Copy link
Author

@dphulkar-msft , hope the fix done for panic close issue is not reverted during this process. Ref - #2832.

Please confirm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants