Skip to content

Commit

Permalink
- Added batch operations support for atomic updates
Browse files Browse the repository at this point in the history
- Implemented safe iterator pattern
- Enhanced performance for bulk operations
- Added comprehensive benchmarking suite
- Improved documentation and examples
  • Loading branch information
jongyunha committed Nov 8, 2024
1 parent 83c1585 commit 0768893
Show file tree
Hide file tree
Showing 10 changed files with 1,043 additions and 102 deletions.
230 changes: 128 additions & 102 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,182 +1,208 @@
# ShrinkableMap

[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Go Reference](https://pkg.go.dev/badge/github.com/jongyunha/shrinkmap.svg)](https://pkg.go.dev/github.com/jongyunha/shrinkmap)
[![Go Report Card](https://goreportcard.com/badge/github.com/jongyunha/shrinkmap)](https://goreportcard.com/report/github.com/jongyunha/shrinkmap)
[![Coverage Status](https://coveralls.io/repos/github/jongyunha/shrinkmap/badge.svg?branch=main)](https://coveralls.io/github/jongyunha/shrinkmap?branch=main)

ShrinkableMap is a high-performance, generic, thread-safe map implementation for Go that automatically manages memory by shrinking its internal storage when items are deleted. It provides a solution to the common issue where Go's built-in maps don't release memory after deleting elements.
ShrinkableMap is a high-performance, generic, thread-safe map implementation for Go that automatically manages memory by shrinking its internal storage when items are deleted. It addresses the common issue where Go's built-in maps don't release memory after deleting elements.

## Features
## 🚀 Features

- 🚀 Generic type support for type-safe operations
- 🔒 Thread-safe implementation with atomic operations
- 📉 Automatic memory shrinking with configurable policies
- ⚙️ Advanced concurrent shrinking behavior
- 📊 Thread-safe performance and error metrics
- 🛡️ Panic recovery and error tracking
- 🔍 Safe state inspection with snapshots
- 🧹 Graceful resource cleanup
- 💪 Production-ready with comprehensive tests
- 🎯 Zero external dependencies
- **Type Safety**
- Generic type support for compile-time type checking
- Type-safe operations for all map interactions

## Installation
- **Performance**
- Optimized concurrent access with minimal locking
- Efficient atomic operations for high throughput
- Batch operations for improved performance

- **Memory Management**
- Automatic memory shrinking with configurable policies
- Advanced concurrent shrinking behavior
- Memory-efficient iterators

- **Reliability**
- Thread-safe implementation
- Panic recovery and error tracking
- Comprehensive metrics collection

- **Developer Experience**
- Safe iteration with snapshot support
- Batch operations for bulk processing
- Clear error reporting and metrics
- Zero external dependencies
- Production-ready with extensive tests

## 📦 Installation

```bash
go get github.com/jongyunha/shrinkmap
```

## Quick Start
## 🔧 Quick Start

```go
package main

import (
"fmt"
"time"
"github.com/jongyunha/shrinkmap"
)

func main() {
// Create a new map with string keys and int values
// Create a new map with default configuration
sm := shrinkmap.New[string, int](shrinkmap.DefaultConfig())

// Ensure cleanup when done
defer sm.Stop()

// Set values
// Basic operations
sm.Set("one", 1)
sm.Set("two", 2)

// Get value
if value, exists := sm.Get("one"); exists {
fmt.Printf("Value: %d\n", value)
}

// Delete value
sm.Delete("one")

// Get current state snapshot
snapshot := sm.Snapshot()
for _, kv := range snapshot {
fmt.Printf("Key: %v, Value: %v\n", kv.Key, kv.Value)
}

// Get metrics including error statistics
metrics := sm.GetMetrics()
fmt.Printf("Total operations: %d\n", metrics.TotalItemsProcessed())
fmt.Printf("Total errors: %d\n", metrics.TotalErrors())
fmt.Printf("Total panics: %d\n", metrics.TotalPanics())
}
```

## Advanced Features
## 💡 Advanced Usage

### Error Tracking and Recovery
### Batch Operations

Monitor and track errors with detailed information:
Efficiently process multiple operations atomically:

```go
metrics := sm.GetMetrics()

// Get error statistics
totalErrors := metrics.TotalErrors()
totalPanics := metrics.TotalPanics()
lastPanicTime := metrics.LastPanicTime()

// Get last error details
if lastError := metrics.LastError(); lastError != nil {
fmt.Printf("Last error: %v\n", lastError.Error)
fmt.Printf("Stack trace: %v\n", lastError.Stack)
fmt.Printf("Time: %v\n", lastError.Timestamp)
batch := shrinkmap.BatchOperations[string, int]{
Operations: []shrinkmap.BatchOperation[string, int]{
{Type: shrinkmap.BatchSet, Key: "one", Value: 1},
{Type: shrinkmap.BatchSet, Key: "two", Value: 2},
{Type: shrinkmap.BatchDelete, Key: "three"},
},
}

// Get error history (last 10 errors)
errorHistory := metrics.ErrorHistory()
for _, err := range errorHistory {
fmt.Printf("Error: %v, Time: %v\n", err.Error, err.Timestamp)
}
// Apply all operations atomically
sm.ApplyBatch(batch)
```

### State Inspection
### Safe Iteration

Safely inspect map state without locking:
Iterate over map contents safely using the iterator:

```go
// Get current state snapshot
// Create an iterator
iter := sm.NewIterator()

// Iterate over all items
for iter.Next() {
key, value := iter.Get()
fmt.Printf("Key: %v, Value: %v\n", key, value)
}

// Or use snapshot for bulk processing
snapshot := sm.Snapshot()
for _, kv := range snapshot {
fmt.Printf("Key: %v, Value: %v\n", kv.Key, kv.Value)
}
```

### Resource Management
### Performance Monitoring

Proper cleanup with graceful shutdown:
Track performance metrics:

```go
// Create map
sm := shrinkmap.New[string, int](shrinkmap.DefaultConfig())
metrics := sm.GetMetrics()
fmt.Printf("Total operations: %d\n", metrics.TotalItemsProcessed())
fmt.Printf("Peak size: %d\n", metrics.PeakSize())
fmt.Printf("Total shrinks: %d\n", metrics.TotalShrinks())
```

// Ensure cleanup
defer sm.Stop()
## 🔍 Configuration Options

// Or stop explicitly when needed
sm.Stop()
```go
config := shrinkmap.Config{
InitialCapacity: 1000,
AutoShrinkEnabled: true,
ShrinkInterval: time.Second,
MinShrinkInterval: time.Second,
ShrinkRatio: 0.5,
CapacityGrowthFactor: 1.5,
MaxMapSize: 1000000,
}
```

## Thread Safety Guarantees
## 🛡️ Thread Safety Guarantees

- All map operations are atomic and thread-safe
- Metrics collection is non-blocking and thread-safe
- Shrinking operations are coordinated to prevent conflicts
- Safe concurrent access from multiple goroutines
- Panic recovery in auto-shrink goroutine
- Thread-safe error tracking and metrics collection
- Safe state inspection with snapshots
- Thread-safe batch operations
- Safe iteration with consistent snapshots
- Coordinated shrinking operations
- Thread-safe metrics collection

## 📊 Performance

## Best Practices
Benchmark results on typical operations (Intel i7-9700K, 32GB RAM):

1. Always ensure proper cleanup:
```
BenchmarkBasicOperations/Sequential_Set-8 5000000 234 ns/op
BenchmarkBasicOperations/Sequential_Get-8 10000000 112 ns/op
BenchmarkBatchOperations/BatchSize_100-8 100000 15234 ns/op
BenchmarkConcurrency/Parallel_8-8 1000000 1123 ns/op
```

## 📝 Best Practices

1. **Resource Management**
```go
sm := shrinkmap.New[string, int](config)
defer sm.Stop() // Ensure auto-shrink goroutine is cleaned up
defer sm.Stop() // Always ensure proper cleanup
```

2. Monitor errors and panics:
2. **Batch Processing**
```go
metrics := sm.GetMetrics()
if metrics.TotalErrors() > 0 {
// Investigate error history
for _, err := range metrics.ErrorHistory() {
log.Printf("Error: %v, Time: %v\n", err.Error, err.Timestamp)
}
}
// Use batch operations for multiple updates
batch := prepareBatchOperations()
sm.ApplyBatch(batch)
```

3. Use snapshots for safe iteration:
3. **Safe Iteration**
```go
snapshot := sm.Snapshot()
for _, kv := range snapshot {
// Process items without holding locks
process(kv.Key, kv.Value)
// Use iterator for safe enumeration
iter := sm.NewIterator()
for iter.Next() {
// Process items safely
}
```

## Version History

- 0.0.2 (ing...)
- Added error tracking and panic recovery
- Added state snapshot functionality
- Added graceful shutdown with Stop()
- Enhanced metrics with error statistics
- Improved resource cleanup
- Added comprehensive error tracking tests

- 0.0.1
- Initial release
- Thread-safe implementation with atomic operations
- Generic type support
- Automatic shrinking with configurable policies
- Comprehensive benchmark suite
- Race condition free guarantee
## 🗓️ Version History

### v0.0.3 (Current)
- Added batch operations support for atomic updates
- Implemented safe iterator pattern
- Enhanced performance for bulk operations
- Added comprehensive benchmarking suite
- Improved documentation and examples

### v0.0.2
- Added error tracking and panic recovery
- Added state snapshot functionality
- Added graceful shutdown
- Enhanced metrics collection
- Improved resource cleanup

### v0.0.1
- Initial release with core functionality
- Thread-safe implementation
- Automatic shrinking support
- Generic type support

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

---
Made with ❤️ by [Jongyun Ha](https://github.com/jongyunha)
47 changes: 47 additions & 0 deletions batch.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package shrinkmap

// BatchOperations provides batch operation capabilities
type BatchOperations[K comparable, V any] struct {
Operations []BatchOperation[K, V]
}

type BatchOperation[K comparable, V any] struct {
Type BatchOpType
Key K
Value V
}

type BatchOpType int

const (
BatchSet BatchOpType = iota
BatchDelete
)

// ApplyBatch applies multiple operations atomically
func (sm *ShrinkableMap[K, V]) ApplyBatch(batch BatchOperations[K, V]) error {
sm.mu.Lock()
defer sm.mu.Unlock()

for _, op := range batch.Operations {
switch op.Type {
case BatchSet:
_, exists := sm.data[op.Key]
sm.data[op.Key] = op.Value
if !exists {
sm.itemCount.Add(1)
sm.updateMetrics(1)
}
case BatchDelete:
if _, exists := sm.data[op.Key]; exists {
delete(sm.data, op.Key)
sm.deletedCount.Add(1)
}
}
}

if sm.config.AutoShrinkEnabled {
go sm.TryShrink()
}
return nil
}
Loading

0 comments on commit 0768893

Please sign in to comment.