Skip to content

Commit

Permalink
Merge pull request #47 from JuliaStrings/aa/bb
Browse files Browse the repository at this point in the history
Use BinaryBuilder-provided binaries, switch CI to GHA
  • Loading branch information
ararslan authored May 24, 2021
2 parents c5b3f9e + d74c52e commit d102671
Show file tree
Hide file tree
Showing 8 changed files with 64 additions and 232 deletions.
49 changes: 49 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: CI
on:
push:
branches: [master]
tags: ["*"]
pull_request:
jobs:
test:
name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
version:
- '1.3'
- '1' # automatically expands to the latest stable 1.x release of Julia
- 'nightly'
os:
- ubuntu-latest
- macOS-latest
- windows-latest
arch:
- x64
- x86
exclude:
- os: macOS-latest
arch: x86
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@v1
with:
version: ${{ matrix.version }}
arch: ${{ matrix.arch }}
- uses: actions/cache@v1
env:
cache-name: cache-artifacts
with:
path: ~/.julia/artifacts
key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
restore-keys: |
${{ runner.os }}-test-${{ env.cache-name }}-
${{ runner.os }}-test-
${{ runner.os }}-
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-runtest@v1
- uses: julia-actions/julia-processcoverage@v1
- uses: codecov/codecov-action@v1
with:
file: lcov.info
8 changes: 1 addition & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
*.jl.cov
*.jl.*.cov
*.jl.mem
deps/builds
deps/downloads
deps/src
deps/usr
deps/deps.jl
deps/build.log
MANIFEST.TOML
Manifest.toml
20 changes: 0 additions & 20 deletions .travis.yml

This file was deleted.

9 changes: 4 additions & 5 deletions Project.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
name = "StringEncodings"
uuid = "69024149-9ee7-55f6-a4c4-859efe599b68"
version = "0.3.3"
version = "0.3.4"

[deps]
BinaryProvider = "b99e7846-7c00-51b0-8f62-c81ae34c0232"
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
Libiconv_jll = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531"

[compat]
BinaryProvider = "0.4.1, 0.5"
julia = "0.7, 1"
Libiconv_jll = "1.16"
julia = "1.3"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
Expand Down
10 changes: 2 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
# StringEncodings

[![Travis CI Build Status](https://travis-ci.org/JuliaStrings/StringEncodings.jl.svg?branch=master)](https://travis-ci.org/JuliaStrings/StringEncodings.jl)
[![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/3gslhfg91isldnvq?svg=true)](https://ci.appveyor.com/project/nalimilan/stringencodings-jl)
[![Coveralls Coverage Status](https://coveralls.io/repos/JuliaStrings/StringEncodings.jl/badge.svg?branch=master&service=github)](https://coveralls.io/github/JuliaStrings/StringEncodings.jl?branch=master)
[![Build status](https://github.com/JuliaStrings/StringEncodings.jl/workflows/CI/badge.svg)](https://github.com/JuliaStrings/StringEncodings.jl/actions?query=workflow%3ACI+branch%3Amaster)
[![Codecov Coverage Status](http://codecov.io/github/JuliaStrings/StringEncodings.jl/coverage.svg?branch=master)](http://codecov.io/github/JuliaStrings/StringEncodings.jl?branch=master)

This Julia package provides support for decoding and encoding texts between multiple character encodings. It is currently based on the iconv interface, and supports all major platforms using either the native iconv support or [GNU libiconv](https://www.gnu.org/software/libiconv/). In the future, native Julia support for major encodings will be added.
This Julia package provides support for decoding and encoding texts between multiple character encodings. It is currently based on the iconv interface, and supports all major platforms using [GNU libiconv](https://www.gnu.org/software/libiconv/). In the future, native Julia support for major encodings will be added.

## Encoding and Decoding Strings
*Encoding* a refers to the process of converting a string (of any `AbstractString` type) to a sequence of bytes represented as a `Vector{UInt8}`. *Decoding* refers to the inverse process.
Expand Down Expand Up @@ -153,7 +151,3 @@ julia> read(s, String) # Decoding happens automatically here
Do not forget to call `close` on `StringEncoder` and `StringDecoder` objects to finish the encoding process. For `StringEncoder`, this function calls `flush`, which writes any characters still in the buffer, and possibly some control sequences (for stateful encodings). For both `StringEncoder` and `StringDecoder`, `close` checks that there are no incomplete sequences left in the input stream, and raise an `IncompleteSequenceError` if that's the case. It will also free iconv resources immediately, instead of waiting for garbage collection.

Conversion currently raises an error if an invalid byte sequence is encountered in the input, or if some characters cannot be represented in the target enconding. It is not yet possible to ignore such characters or to replace them with a placeholder.

## Notes on Installation on Linux OS

Most Linux distributions provide `iconv` functionalities as part of the base operating system library `glibc`. In normal circumstances, no additional installation of `libiconv` should be required. If you observe such a behavior on your operating system, file an issue with OS details.
42 changes: 0 additions & 42 deletions appveyor.yml

This file was deleted.

124 changes: 0 additions & 124 deletions deps/build.jl

This file was deleted.

34 changes: 8 additions & 26 deletions src/StringEncodings.jl
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,7 @@

module StringEncodings

using Libdl

# Load in `deps.jl`, complaining if it does not exist
const depsjl_path = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")
if !isfile(depsjl_path)
error("iconv not installed properly, run Pkg.build(\"StringEncodings\"), restart Julia and try again")
end
include(depsjl_path)

# Module initialization function
function __init__()
# Always check dependencies from `deps.jl`
check_deps()
end
using Libiconv_jll

using Base.Libc: errno, strerror, E2BIG, EINVAL, EILSEQ

Expand Down Expand Up @@ -75,13 +62,13 @@ show(io::IO, exc::T) where {T<:Union{IncompleteSequenceError,OutputBufferError}}

function iconv_close(cd::Ptr{Nothing})
if cd != C_NULL
ccall((iconv_close_s, libiconv), Cint, (Ptr{Nothing},), cd) == 0 ||
ccall((:libiconv_close, libiconv), Cint, (Ptr{Nothing},), cd) == 0 ||
throw(IConvError("iconv_close"))
end
end

function iconv_open(tocode::String, fromcode::String)
p = ccall((iconv_open_s, libiconv), Ptr{Nothing}, (Cstring, Cstring), tocode, fromcode)
p = ccall((:libiconv_open, libiconv), Ptr{Nothing}, (Cstring, Cstring), tocode, fromcode)
if p != Ptr{Nothing}(-1)
return p
elseif errno() == EINVAL
Expand Down Expand Up @@ -144,7 +131,7 @@ function iconv!(cd::Ptr{Nothing}, inbuf::Vector{UInt8}, outbuf::Vector{UInt8},
inbytesleft_orig = inbytesleft[]
outbytesleft[] = BUFSIZE

ret = ccall((iconv_s, libiconv), Csize_t,
ret = ccall((:libiconv, libiconv), Csize_t,
(Ptr{Nothing}, Ptr{Ptr{UInt8}}, Ref{Csize_t}, Ptr{Ptr{UInt8}}, Ref{Csize_t}),
cd, inbufptr, inbytesleft, outbufptr, outbytesleft)

Expand Down Expand Up @@ -176,7 +163,7 @@ function iconv_reset!(s::Union{StringEncoder, StringDecoder})

s.outbufptr[] = pointer(s.outbuf)
s.outbytesleft[] = BUFSIZE
ret = ccall((iconv_s, libiconv), Csize_t,
ret = ccall((:libiconv, libiconv), Csize_t,
(Ptr{Nothing}, Ptr{Ptr{UInt8}}, Ref{Csize_t}, Ptr{Ptr{UInt8}}, Ref{Csize_t}),
s.cd, C_NULL, C_NULL, s.outbufptr, s.outbytesleft)

Expand Down Expand Up @@ -278,9 +265,6 @@ read from `stream` into text in the encoding `to`. Calling `close` on the
stream does not close `stream`.
`to` and `from` can be specified either as a string or as an `Encoding` object.
Note that some implementations may accept invalid sequences
in the input data without raising an error.
"""
function StringDecoder(stream::IO, from::Encoding, to::Encoding=enc"UTF-8")
cd = iconv_open(String(to), String(from))
Expand Down Expand Up @@ -542,7 +526,7 @@ encode(s::AbstractString, enc::AbstractString) = encode(s, Encoding(enc))

function test_encoding(enc::String)
# We assume that an encoding is supported if it's possible to convert from it to UTF-8:
cd = ccall((iconv_open_s, libiconv), Ptr{Nothing}, (Cstring, Cstring), enc, "UTF-8")
cd = ccall((:libiconv_open, libiconv), Ptr{Nothing}, (Cstring, Cstring), enc, "UTF-8")
if cd == Ptr{Nothing}(-1)
return false
else
Expand All @@ -555,15 +539,13 @@ end
encodings()
List all encodings supported by `encode`, `decode`, `StringEncoder` and `StringDecoder`
(i.e. by the current iconv implementation).
(i.e. by GNU libiconv).
Note that encodings typically appear several times under different names.
In addition to the encodings returned by this function, the empty string (i.e. `""`)
is equivalent to the encoding of the current locale.
Some implementations may support even more encodings: this can be checked by attempting
a conversion. In theory, it is not guaranteed that all conversions between all pairs of encodings
are possible; but this is the case with all reasonable implementations.
Even more encodings may be supported: this can be checked by attempting a conversion.
"""
function encodings()
filter(test_encoding, encodings_list)
Expand Down

2 comments on commit d102671

@ararslan
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/37388

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.3.4 -m "<description of version>" d102671db9e4636b699ad2ffe07226d9e15a739e
git push origin v0.3.4

Please sign in to comment.