Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compression and JustSaying 8.0 Changes #1525

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
a58a62b
Initial subscription compression functionality
slang25 May 9, 2024
cc7d4d3
WIP - more compression changes
slang25 Jun 12, 2024
2d3c495
Add publisher compression support
slang25 Jun 26, 2024
426030d
Address more feedback
slang25 Jun 27, 2024
e9a8a77
Further compression development
slang25 Jul 8, 2024
ddead74
Add some extra null handling
slang25 Jul 8, 2024
37eaa73
Tweaked behaviour of CompressMessageIfNeeded
slang25 Jul 8, 2024
09413c3
Add more xml doc comments
slang25 Jul 8, 2024
ab075b0
Add null check and remove binary breaking change
slang25 Jul 8, 2024
d1dd060
Work on public api
slang25 Jul 8, 2024
fff6506
Add back API which was made private
slang25 Jul 8, 2024
aa326ce
Fix up API files
slang25 Jul 8, 2024
c9ec0f9
Add tests and fixes
slang25 Jul 8, 2024
a4ef78c
Move missing encoding check
slang25 Jul 8, 2024
619b199
Add extra validation
slang25 Jul 8, 2024
ce4b6d4
Add unit tests
slang25 Jul 8, 2024
b3e5222
Rename DefaultCompressionOptions
slang25 Jul 8, 2024
82a0bca
More work
slang25 Jul 9, 2024
cd4f61b
WIP
slang25 Jul 9, 2024
7c18a0d
More compression refactoring
slang25 Jul 11, 2024
df43e02
Tidy up
slang25 Jul 12, 2024
8c23f2f
Fix CI
slang25 Jul 12, 2024
b0ea0ef
Compression WIP
slang25 Aug 22, 2024
4d732a0
More compression work
slang25 Aug 26, 2024
d888b5a
Add RawMessageDelivery support
slang25 Aug 27, 2024
c593ef6
Fix more tests
slang25 Sep 4, 2024
3657d8d
Add more tests and cleanup code
slang25 Oct 3, 2024
d644e48
More tidy up
slang25 Oct 3, 2024
6c04948
More tidy up
slang25 Oct 3, 2024
cb8e0fd
More tidy up
slang25 Oct 3, 2024
fe1d325
More tidy up
slang25 Oct 3, 2024
140ba8e
Merge branch 'main' into refactored-compression
slang25 Oct 3, 2024
8ca09ad
More tidy up
slang25 Oct 3, 2024
d2b2a8f
Apply some feedback
slang25 Oct 4, 2024
fbe31d9
Apply some feedback
slang25 Oct 4, 2024
64e0d21
Apply some feedback
slang25 Oct 4, 2024
ee7febf
Apply some feedback
slang25 Oct 4, 2024
ef110b8
Apply some feedback
slang25 Oct 4, 2024
8de4441
More changes
slang25 Oct 7, 2024
3872c83
Fix json encoding issues
slang25 Oct 7, 2024
7e016e6
Tidy up a few bits
slang25 Oct 8, 2024
d549a5c
Apply some feedback
slang25 Oct 8, 2024
af27eae
More refactoring
slang25 Oct 9, 2024
c2e8f0a
Merge remote-tracking branch 'upstream/main' into refactored-compression
slang25 Oct 11, 2024
e1daeee
Fix test
slang25 Oct 11, 2024
0ade76f
Update LocalSqsSnsMessaging and fix flakey test
slang25 Oct 15, 2024
e1a257c
Apply feedback
slang25 Oct 15, 2024
6e76da9
Support overriding subject
slang25 Oct 18, 2024
07d53d3
Make tests more resiliant
slang25 Oct 21, 2024
6830ed9
Improve relaibility of tests
slang25 Oct 23, 2024
1f69d00
Improve relaibility of tests
slang25 Oct 23, 2024
d330c1f
More test tweaks
slang25 Oct 23, 2024
caad08d
Improve relaibility of tests
slang25 Oct 23, 2024
57694a6
Improve relaibility of tests
slang25 Oct 23, 2024
fa76dae
Improve relaibility of tests
slang25 Oct 25, 2024
c7c782d
Improve relaibility of tests
slang25 Oct 26, 2024
aab0086
Improve relaibility of tests
slang25 Oct 27, 2024
4f0b1ae
Reduce unnecessary exceptions
slang25 Oct 27, 2024
d58dc2c
Run tests in parallel
slang25 Oct 27, 2024
850202e
Revert "Run tests in parallel"
slang25 Oct 27, 2024
173091a
Make PublishDestinationType internal
slang25 Oct 28, 2024
e77b78a
Revert samples change
slang25 Nov 6, 2024
ed96100
Extend message attribute handling
slang25 Nov 6, 2024
01bbbd3
Add improved message size logic and fixed duplicate lines
slang25 Nov 6, 2024
38157f0
Revert localstack ports for samples
slang25 Nov 6, 2024
4a05bd3
Merge main
slang25 Nov 6, 2024
dd12c60
Merge remote-tracking branch 'upstream/main' into refactored-compression
slang25 Nov 6, 2024
bf35392
Revert SDK version
slang25 Nov 6, 2024
4054945
Add integration test
slang25 Nov 7, 2024
5a39f7c
Update src/JustSaying/AwsTools/MessageHandling/PublishMessageConverte…
slang25 Nov 7, 2024
b850c1c
Move ParseMessageAttribute to usage
slang25 Nov 7, 2024
6626806
Update tests/JustSaying.Extensions.DependencyInjection.StructureMap.T…
slang25 Nov 7, 2024
99e6efd
Update src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConvert…
slang25 Nov 7, 2024
474654a
Update tests/JustSaying.UnitTests/Messaging/Compression/GzipMessageBo…
slang25 Nov 7, 2024
009886b
Reduce boilerplate
slang25 Nov 7, 2024
01c124f
Add cancellation token
slang25 Nov 7, 2024
96a1a79
Reduce default compression size
slang25 Nov 7, 2024
0e7372a
Rename types
slang25 Nov 8, 2024
43c59c4
Use STJ in tests
slang25 Nov 8, 2024
8212cbb
Add TODO note for adding back LocalStack tests
slang25 Nov 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
var decompressor = _compressionRegistry.GetCompression(contentEncoding.StringValue);
if (decompressor is null)
{
throw new InvalidOperationException($"Compression encoding '{contentEncoding.StringValue}' is not registered.");

Check warning on line 50 in src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs

View check run for this annotation

Codecov / codecov/patch

src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs#L50

Added line #L50 was not covered by tests
}

body = decompressor.Decompress(body);
Expand Down Expand Up @@ -76,17 +76,31 @@
var dataType = property.Value.GetProperty("Type").GetString();
var dataValue = property.Value.GetProperty("Value").GetString();

attributes.Add(property.Name, MessageAttributeParser.Parse(dataType, dataValue));
attributes.Add(property.Name, ParseMessageAttribute(dataType, dataValue));
}

return new MessageAttributes(attributes);
}

public static MessageAttributeValue ParseMessageAttribute(string dataType, string dataValue)
slang25 marked this conversation as resolved.
Show resolved Hide resolved
{
// Check for a prefix instead of an exact match as SQS supports custom-type labels, or example, "Binary.gif".
// See https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes.
bool isBinary = dataType?.StartsWith("Binary", StringComparison.Ordinal) is true;

return new()
{
DataType = dataType,
StringValue = !isBinary ? dataValue : null,
BinaryValue = isBinary ? Convert.FromBase64String(dataValue) : null
};
}

private static MessageAttributes GetRawMessageAttributes(Amazon.SQS.Model.Message message)
{
if (message.MessageAttributes is null)
{
return new MessageAttributes();

Check warning on line 103 in src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs

View check run for this annotation

Codecov / codecov/patch

src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs#L103

Added line #L103 was not covered by tests
}

Dictionary<string, MessageAttributeValue> rawAttributes = new ();
Expand All @@ -95,7 +109,7 @@
{
var dataType = messageMessageAttribute.Value.DataType;
var dataValue = messageMessageAttribute.Value.StringValue;
rawAttributes.Add(messageMessageAttribute.Key, MessageAttributeParser.Parse(dataType, dataValue));
rawAttributes.Add(messageMessageAttribute.Key, ParseMessageAttribute(dataType, dataValue));
}

return new MessageAttributes(rawAttributes);
Expand All @@ -113,7 +127,7 @@
var utf8JsonReader = new Utf8JsonReader(Encoding.UTF8.GetBytes(body));
if (!JsonDocument.TryParseValue(ref utf8JsonReader, out var jsonDocument))
{
return false;

Check warning on line 130 in src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs

View check run for this annotation

Codecov / codecov/patch

src/JustSaying/AwsTools/MessageHandling/ReceivedMessageConverter.cs#L130

Added line #L130 was not covered by tests
}

using (jsonDocument)
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ namespace JustSaying;
public class WhenUsingStructureMap(ITestOutputHelper outputHelper)
{
private ITestOutputHelper OutputHelper { get; } = outputHelper;
private InMemoryAwsBus InMemoryAwsBus { get; } = new ();
private InMemoryAwsBus InMemoryAwsBus { get; } = new();

[AwsFact]
public async Task Can_Create_Messaging_Bus_Fluently_For_A_Queue()
Expand Down
Loading