Adding rules in code produces different results and logs than the rules I added directly from the command line #247

xlango · 2023-11-09T10:33:12Z

When i used command lines
{
nft add table ip filter
nft add chain ip filter output { type filter hook output priority 0; }
nft add set filter ipSet { type ipv4_addr ; flags interval;}
nft add element ip filter ipSet {10.34.11.179}
nft add set filter portSet { type inet_service ; flags interval;}
nft add element ip filter portSet {1234}
nft add rule ip filter output ip daddr @ipSet tcp dport @portSet counter log drop
}

to add a rule related to set, it worked correctly.
But when i used nftables-main to add a similar rule, it blocked tcp flow to "127.0.0.1" too.
The code is :
func main() {
c, err := nftables.New()
if err != nil {
return
}

c.FlushRuleset()

filter := c.AddTable(&nftables.Table{
	Family: nftables.TableFamilyIPv4,
	Name:   "filter",
})

input := c.AddChain(&nftables.Chain{
	Name:     "output",
	Hooknum:  nftables.ChainHookOutput,
	Priority: nftables.ChainPriorityFilter,
	Table:    filter,
	Type:     nftables.ChainTypeFilter,
})
ipSet := &nftables.Set{
	Name:"ipSet",
	Table:     filter,
	Interval:      true,
	Concatenation: true,
	KeyType:   nftables.TypeIPAddr,
}
if err := c.AddSet(ipSet, []nftables.SetElement{
	{
		Key: []byte(net.ParseIP("10.34.11.179").To4()),
		KeyEnd: []byte(net.ParseIP("10.34.11.180").To4()),
	},
}); err != nil {
	return
}

portSet := &nftables.Set{
	Name:"portSet",
	Table:     filter,
	Interval:      true,
	Concatenation: true,
	KeyType:    nftables.TypeInetService,
}

if err := c.AddSet(portSet, []nftables.SetElement{
	{
		Key: binaryutil.BigEndian.PutUint16(1234),
		KeyEnd:binaryutil.BigEndian.PutUint16(1235),
	},
}); err != nil {
	return
}

c.AddRule(&nftables.Rule{
	Table: filter,
	Chain: input,
	Exprs: []expr.Any{
		&expr.Payload{
			DestRegister: 1,
			Base:         expr.PayloadBaseNetworkHeader,
			Offset:       16,
			Len:          4,
		},

		&expr.Lookup{
			SourceRegister: 1,
			SetName:        ipSet.Name,
			SetID:          ipSet.ID,
		},

		&expr.Meta{Key: expr.MetaKeyL4PROTO, Register: 1},
		&expr.Cmp{
			Op:       expr.CmpOpEq,
			Register: 1,
			Data:     []byte{unix.IPPROTO_TCP},
		},

		&expr.Payload{
			DestRegister: 1,
			Base:         expr.PayloadBaseTransportHeader,
			Offset:       2,
			Len:          2,
		},

		&expr.Lookup{
			SourceRegister: 1,
			SetName:        portSet.Name,
			SetID:          portSet.ID,
		},

		&expr.Counter{},
		&expr.Log{},
		&expr.Verdict{
			Kind: expr.VerdictDrop,
		},
	},
})
if err := c.Flush(); err != nil {
	return
}

}

log

The text was updated successfully, but these errors were encountered:

turekt · 2023-11-15T23:43:40Z

Hi @xlango,

your code does not replicate netlink messages sent by the nft command.

Can you test the following code and let me know if it works?

package main

import (
	"github.com/google/nftables"
	"github.com/google/nftables/binaryutil"
	"github.com/google/nftables/expr"
	"golang.org/x/sys/unix"
	"net"
)

func main() {
	c, err := nftables.New()
	if err != nil {
		panic(err)
	}
	c.FlushRuleset()

	filter := c.AddTable(&nftables.Table{
		Family: nftables.TableFamilyIPv4,
		Name:   "filter",
	})

	input := c.AddChain(&nftables.Chain{
		Name:     "output",
		Hooknum:  nftables.ChainHookOutput,
		Priority: nftables.ChainPriorityFilter,
		Table:    filter,
		Type:     nftables.ChainTypeFilter,
	})
	ipSet := &nftables.Set{
		Name:     "ipSet",
		Table:    filter,
		Interval: true,
		KeyType:  nftables.TypeIPAddr,
	}
	if err := c.AddSet(ipSet, nil); err != nil {
		panic(err)
	}
	if err := c.SetAddElements(ipSet, []nftables.SetElement{
		{
			Key:         []byte{0x00, 0x00, 0x00, 0x00},
			IntervalEnd: true,
		},
		{
			Key: []byte(net.ParseIP("10.34.11.179").To4()),
		},
		{
			Key:         []byte(net.ParseIP("10.34.11.180").To4()),
			IntervalEnd: true,
		},
	}); err != nil {
		panic(err)
	}

	portSet := &nftables.Set{
		Name:     "portSet",
		Table:    filter,
		Interval: true,
		KeyType:  nftables.TypeInetService,
	}

	if err := c.AddSet(portSet, nil); err != nil {
		panic(err)
	}
	if err := c.SetAddElements(portSet, []nftables.SetElement{
		{
			Key:         []byte{0x00, 0x00},
			IntervalEnd: true,
		},
		{
			Key: binaryutil.BigEndian.PutUint16(1234),
		},
		{
			Key:         binaryutil.BigEndian.PutUint16(1235),
			IntervalEnd: true,
		},
	}); err != nil {
		panic(err)
	}

	c.AddRule(&nftables.Rule{
		Table: filter,
		Chain: input,
		Exprs: []expr.Any{
			&expr.Payload{
				DestRegister: 1,
				Base:         expr.PayloadBaseNetworkHeader,
				Offset:       16,
				Len:          4,
			},

			&expr.Lookup{
				SourceRegister: 1,
				SetName:        ipSet.Name,
				SetID:          ipSet.ID,
			},

			&expr.Meta{Key: expr.MetaKeyL4PROTO, Register: 1},
			&expr.Cmp{
				Op:       expr.CmpOpEq,
				Register: 1,
				Data:     []byte{unix.IPPROTO_TCP},
			},

			&expr.Payload{
				DestRegister: 1,
				Base:         expr.PayloadBaseTransportHeader,
				Offset:       2,
				Len:          2,
			},

			&expr.Lookup{
				SourceRegister: 1,
				SetName:        portSet.Name,
				SetID:          portSet.ID,
			},

			&expr.Counter{},
			&expr.Log{},
			&expr.Verdict{
				Kind: expr.VerdictDrop,
			},
		},
	})
	if err := c.Flush(); err != nil {
		panic(err)
	}
}

I hope that this resolves your issue.

realsyy · 2023-11-16T04:52:10Z

@turekt
Thanks, it worked now.But i wonder why should i add a rule set element like
"{
Key: []byte{0x00, 0x00, 0x00, 0x00},
IntervalEnd: true,
}" ?

turekt · 2023-11-16T14:38:31Z

Hi @realsyy,

when analysing mnl debug information, you will see that this set element is added by nft (reduced output for brevity):

$ nft --debug=mnl add element ip filter ipSet {10.34.11.179}
...
----------------	------------------
|  0000000120  |	| message length |
| 02572 | R--- |	|  type | flags  |
|  0000000001  |	| sequence number|
|  0000000000  |	|     port ID    |
----------------	------------------
| 02 00 00 00  |	|  extra header  |
|00011|--|00001|	|len |flags| type|
| 66 69 6c 74  |	|      data      |	 f i l t
| 65 72 00 00  |	|      data      |	 e r    
|00010|--|00002|	|len |flags| type|
| 69 70 53 65  |	|      data      |	 i p S e
| 74 00 00 00  |	|      data      |	 t      
|00008|--|00004|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00068|N-|00003|	|len |flags| type|
|00024|N-|00001|	|len |flags| type|
|00008|--|00003|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 00 00 00 00  |	|      data      |	        
|00016|N-|00002|	|len |flags| type|
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 0a 22 0b b3  |	|      data      |	   "    
|00024|N-|00003|	|len |flags| type|
|00008|--|00003|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 0a 22 0b b4  |	|      data      |	   "    
----------------	------------------
...

Dissection of the netlink packet (nft_set_elem_list_attributes) as follows:

11 -- 1 (NFTA_SET_ELEM_LIST_TABLE)    | filter\x00\x00    |    --> table name (filter)
10 -- 2 (NFTA_SET_ELEM_LIST_SET)      | ipSet\x00\x00\x00 |    --> set name (ipSet)
 8 -- 4 (NFTA_SET_ELEM_LIST_SET_ID)   | \x00\x00\x00\x01  |    --> set ID (1)
68 N- 3 (NFTA_SET_ELEM_LIST_ELEMENTS) |                   |    --> nested, nft_set_elem_attributes struct
24 N- 1                               |                   |    --> nested, first element
 8 -- 3 (NFTA_SET_ELEM_FLAGS)         | \x00\x00\x00\x01  |    --> flag NFT_SET_ELEM_INTERVAL_END
12 N- 1                               |                   |    --> nested
 8 -- 1 (NFTA_SET_ELEM_KEY)           | \x00\x00\x00\x00  |    --> key, 4 bytes, value 0
16 N- 2                               |                   |    --> nested, second element
12 N- 1                               |                   |    --> nested
 8 -- 1 (NFTA_SET_ELEM_KEY)           | \x0a\x22\x0b\xb3  |    --> key, 4 bytes, value IP 10.34.11.179
24 N- 3                               |                   |    --> nested, third element
 8 -- 3 (NFTA_SET_ELEM_FLAGS)         | \x00\x00\x00\x01  |    --> flag NFT_SET_ELEM_INTERVAL_END
12 N- 1                               |                   |    --> nested
 8 -- 1 (NFTA_SET_ELEM_KEY)           | \x0a\x22\x0b\xb4  |    --> key, 4 bytes, value IP 10.34.11.180

Same structure is observed when setting debug to netlink:

$ nft --debug=netlink add element ip filter ipSet {10.34.11.179}
ip filter @ipSet
	element b40b220a  : 1 [end]
	element b30b220a  : 0 [end]
	element 00000000  : 1 [end]
ipSet filter 0
	element b30b220a  : 0 [end]	element b40b220a  : 1 [end]

As to why this additional element is added by nftables, my guess would be due to use of segment trees in nftables intervals.

I don't know for sure but code shows a call to set_to_intervals when elements are being added via do_add_elements function here: https://git.netfilter.org/nftables/tree/src/rule.c?id=2f1050a6b30b41f4125ab6f0da7ea5255090ccce#n1431

static int do_add_elements(struct netlink_ctx *ctx, struct cmd *cmd,
			   uint32_t flags)
{
	struct expr *init = cmd->expr;
	struct set *set = cmd->elem.set;

	if (set_is_non_concat_range(set) &&
	    set_to_intervals(set, init, true) < 0)
		return -1;

	return __do_add_elements(ctx, cmd, set, init, flags);
}

The set_to_intervals checks whether a set requires "first segment": https://git.netfilter.org/nftables/tree/src/intervals.c?id=2f1050a6b30b41f4125ab6f0da7ea5255090ccce#n673:

...
if (!prev && segtree_needs_first_segment(set, init, add) &&
    mpz_cmp_ui(elem->key->left->value, 0)) {
...

Condition whether the first segment should be added is met because the set exists and there are no elements yet: https://git.netfilter.org/nftables/tree/src/intervals.c?id=2f1050a6b30b41f4125ab6f0da7ea5255090ccce#n631

static bool segtree_needs_first_segment(const struct set *set,
					const struct expr *init, bool add)
{
	if (add && !set->root) {
		/* Add the first segment in four situations:
		 *
		 * 1) This is an anonymous set.
		 * 2) This set exists and it is empty.
		 * 3) New empty set and, separately, new elements are added.
		 * 4) This set is created with a number of initial elements.
		 */
		if ((set_is_anonymous(set->flags)) ||
		    (set->init && set->init->size == 0) ||
		    (set->init == NULL && init) ||
		    (set->init == init)) {
			return true;
		}
...

The content of this first segment is added inside the if block. Its value is set to 0 and flags to INTERVAL_END: https://git.netfilter.org/nftables/tree/src/intervals.c?id=2f1050a6b30b41f4125ab6f0da7ea5255090ccce#n680

...
// sets value to 0
mpz_set(expr->value, p);
// expr set as root
root = set_elem_expr_alloc(&internal_location, expr);
if (i->etype == EXPR_MAPPING) {
	root = mapping_expr_alloc(&internal_location,
	  root,
	  expr_get(i->right));
}
// adds flag for interval end
root->flags |= EXPR_F_INTERVAL_END;
list_add(&root->list, &intervals);
...

realsyy · 2023-11-20T06:12:55Z

turekt · 2023-11-20T14:48:08Z

@realsyy Once you obtain the debug messages, you need to check the source code to dissect packets.

Using your case as an example.

Obtain the debug message:

...
----------------	------------------
|  0000000120  |	| message length |
| 02572 | R--- |	|  type | flags  |
|  0000000001  |	| sequence number|
|  0000000000  |	|     port ID    |
----------------	------------------
| 02 00 00 00  |	|  extra header  |
|00011|--|00001|	|len |flags| type|
| 66 69 6c 74  |	|      data      |	 f i l t
| 65 72 00 00  |	|      data      |	 e r    
|00010|--|00002|	|len |flags| type|
| 69 70 53 65  |	|      data      |	 i p S e
| 74 00 00 00  |	|      data      |	 t      
|00008|--|00004|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00068|N-|00003|	|len |flags| type|
|00024|N-|00001|	|len |flags| type|
|00008|--|00003|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 00 00 00 00  |	|      data      |	        
|00016|N-|00002|	|len |flags| type|
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 0a 22 0b b3  |	|      data      |	   "    
|00024|N-|00003|	|len |flags| type|
|00008|--|00003|	|len |flags| type|
| 00 00 00 01  |	|      data      |	        
|00012|N-|00001|	|len |flags| type|
|00008|--|00001|	|len |flags| type|
| 0a 22 0b b4  |	|      data      |	   "    
----------------	------------------
...

Use nf_tables.h from either netfilter or linux source code: https://git.netfilter.org/libnftnl/tree/include/linux/netfilter/nf_tables.h

The 2572 as message type marks the NFT_MSG type:

>>> bin(2572)
'0b101000001100'

Checking for message types in source code: https://git.netfilter.org/libnftnl/tree/include/linux/netfilter/nf_tables.h?id=3eaa940bc33a3186dc7ba1e30640ec79b5f261b9#n101, index in the enum corresponds to its value, so NFT_MSG_NEWSETELEM is 12:

>>> bin(12)
'0b1100'

The last 8 bits in the type are the same, so we can conclude that this is a message for adding a new set element. The meaning of the first 4 bits in the message type value (2560) is for you to try and conclude on your own.

Now that we know that this is a message for adding a new set element, the comment above the nf_tables_msg_types enum reveals:

 * @NFT_MSG_NEWSETELEM: create a new set element (enum nft_set_elem_attributes)

Moving to the appropriate struct: https://git.netfilter.org/libnftnl/tree/include/linux/netfilter/nf_tables.h?id=3eaa940bc33a3186dc7ba1e30640ec79b5f261b9#n422

The enum index corresponds to its integer value, so NFTA_SET_ELEM_KEY is 1, NFTA_SET_ELEM_DATA is 2 and so on.

I'm leaving further dissection of the above packet to you as an exercise.

realsyy · 2023-11-24T01:57:02Z

@turekt Yes, i can analyze a debug message as you do by doing manual work,i thought there may be a tool like nft or any other binary program which can analyze a debug message itself, all you do is just type a bash command.Thanks very much for your guidance!

jcaamano mentioned this issue Jun 4, 2024

port management-port-related iptables code to nftables ovn-kubernetes/ovn-kubernetes#3709

Merged

Zempashi mentioned this issue Jul 12, 2024

firewall ruleset with sequential port list results in blocked port siderolabs/talos#9009

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding rules in code produces different results and logs than the rules I added directly from the command line #247

Adding rules in code produces different results and logs than the rules I added directly from the command line #247

xlango commented Nov 9, 2023

turekt commented Nov 15, 2023

realsyy commented Nov 16, 2023

turekt commented Nov 16, 2023

realsyy commented Nov 20, 2023 •

edited

Loading

turekt commented Nov 20, 2023

realsyy commented Nov 24, 2023

Adding rules in code produces different results and logs than the rules I added directly from the command line #247

Adding rules in code produces different results and logs than the rules I added directly from the command line #247

Comments

xlango commented Nov 9, 2023

turekt commented Nov 15, 2023

realsyy commented Nov 16, 2023

turekt commented Nov 16, 2023

realsyy commented Nov 20, 2023 • edited Loading

turekt commented Nov 20, 2023

realsyy commented Nov 24, 2023

realsyy commented Nov 20, 2023 •

edited

Loading