How about you delete all the queue rules altogether and test? Is it still spiking on ipv6 only?

I setup an AWS instance that do nothing.
IPv4: 43.216.194.71
IPv6: 2406:da10:8a99:4e02:27ef:8e72:86c7:f5a2

Can you ping plotter with this 2 address simultaneously?

Ping it like a minute without upload.
Then start the upload.

I just want to get an idea about the congestion behavior.

EDIT:
This TM really lewat one la. Last time my FSU. Appointment 10am, they came 3:30pm.

This post has been edited by kwss: Feb 15 2025, 12:01 PM

OK, I can clearly see a ladder pattern in the IPv6 latency.

This is a local congestion but why only IPv6 right?
If you dig back some of my old post in Unifi thread, I talk about how TM cheat speedtest.

They are using the same method to cheat bufferbloat test here.

What happen:
They configure a special queue targeting certain packet. So instead of having a sparse queue which fairly apply to all packet, they create special queue that selectively target IPv4 ICMP packet.
However, they forgot to create the same rule for IPv6.

Now the number might look nice in your ping plotter, but the real packet performance is what you are getting now.

Why queue "workaround" the problem is due to their HQoS.
1gig package has a higher priority traffic class than other traffic. But if you overshoot your upload limit, it ends up queueing.

Remember this is not a guaranteed situation. During peak usage, there is only so much buffer left and they will all be dropped. So HQoS in that case act as a policer instead of a shaper.

So yea, keep bitching to TM until they upgrade their shitty pipe. But it won't happen overnight.

Or keep escalating until someone create a rule for IPv6 so it will "looks good"

This post has been edited by kwss: Feb 15 2025, 12:30 PM

Yup, depending on which VPN you use, they might use different MPLS circuit that's not congested and it will work.

EDIT:
I shutdown my AWS instance already.

This post has been edited by kwss: Feb 15 2025, 12:33 PM

Yeap, so much better. So I would say highly likely is due to local congestion.

When the TM tech come, why don't you straight away frame him for local congestion and see his reaction? LOL.

Actually the packet loss in the middle are all loss of TTL Time Exceeded packet. I don't know if those are ASIC data plane but since those are Google's router, I assume they are.

The actual packet loss to your endpoint is only 0.5%.

Still doesn't explain why your Mikrotik loss so much TTL Time Exceeded packet, that remains a mystery.

Post again after they visit.
Remember to also try to get rid of all those error reported by the ONU.

Not possible.
This one works on T-CONT. It won't selectively burn IPv6. In fact it is not even aware of what kind of packet.

I won't deny a lot of ISP cheat only IPv4 packet due to gaming. But that is at upstream, not at L2

This post has been edited by kwss: Feb 15 2025, 01:05 PM

Your observation is correct but technically it is the wrong understanding.
When you are using a "correct VPN", you are actually using another circuit that is not congested.
The term "correct VPN" here means the BNG will egress that packet to a different, uncongested MPLS circuit.

You can use the wrong VPN which end up in the congested circuit and still see terrible result.
How to choose the correct or incorrect VPN? Trial and error + pure luck

EDIT:
Ping and traceroute also don't necessary translate directly to packet forwarding performance. This is a very important fact to remember. During packet forwarding, they traverse only data plane. If it is ASIC, then it literally fly by at wirespeed.
However, any ICMP generation will cause the data plane to interrupt and copy the packet to control plane, which is just general purpose CPU and not real time.

This post has been edited by kwss: Feb 15 2025, 01:18 PM

I know. I answered it in my edit while you posting again, lol

Hence performance testing must always be done end to end, using userland software like iperf3, TRex etc.

This post has been edited by kwss: Feb 15 2025, 01:24 PM

I already expected that's kind of the outcome.
But you didn't even borrow some ONU to test for error and ask for a swap?
I expect that's the minimum you ask for.

You should just swap your old Alcatel with the ZTE. At least you can login and get many more stuff out of it. The Alcatel is crazy locked down.
Did you ask him about the OLT you are using and potentially switch you to a different line card?

You should ask a tech to do what they can help you do. Configuring QoS and all those are not their daily job scope.

go626201
I more or less can reproduce your issue by blasting it with high packet per second (pps).
As you know forwarding / routing performance is based on pps, not bandwidth.

On my setup, anything more than 160k pps will stall the ONU and start missing packet. Mine is all 1gig setup. I am using 2gig service as a symmetric 1gig.



With big packet (1400 bytes), 160k pps is equivalent to 1.7Gbps so you probably will pass speedtest.

All test performed using iperf3. Command like below:
iperf3 -c <server> -u -l 16 -b 100M

-l specify the packet length.

You can also use -P to specify number of parallel thread.

Lastly, not exactly sure if this is your issue but this is the only way I can reproduce the problem.


Attached thumbnail(s)

Why partially fixed? Looks like fully fixed for me. Any problem that I am not seeing?
You are uploading at full 500 meg and latency is all green without queue. 0 packet loss to BNG.

I am not familiar with how smokeping calculate the average. Wait for a week and see if the graph correct itself?
I suspect it just trend higher because you keep testing upload.
What idle latency did you get from ping plotter / speedtest if you stop testing your upload?

People, I got TM VoIP working on Android phone again.
This is despite they closed the loophole by simply disabling VLAN 400 from being forwarded to LAN 1.

The technique is simple, and works even better than last time because we no longer need to dstnat to the Android phone.
The reason is because it was double NAT. Once on the Mikrotik, the second time on the ONU. Yes, the ONU actually perform NAT on the VoIP VLAN.

First add 2 static route in the ONU. Assuming:
192.168.88.0/24 = Your daily usage network
192.168.1.1 = ONU IP Address
192.168.1.10 = Router IP Address

If your setup is different, adjust accordingly. Here is the screenshot of the ONU static routing



After you are done, add a static route to your router. It will work on any router this time since no NAT is required.



Android phone configuration unchanged. Just make call.

it is even better now. If you can VPN home, you can make call via TM number. Previously this cannot be done due to the need for dstnat.

Hi, I am using the Android default Phone app for this. Just tested and it is still working.
The whole ONU NAT for VLAN 400 is the default, at least for me using ZTE F620.

The ONU VoIP stack works like this:
1. VLAN 400 comes into the ONU
2. ONU will pipe the VLAN to CPU and run a DHCP client on it. This process also perform NAT on it.
3. Instead of adding a default route, it will act like a separate VRF. So TM will have all their VLAN starts with 10.x.x.x and they all works because of this.
4. The VoIP subsystem will then attach to this "VRF". That's why in the ONU, you can use Ping or Traceroute tool and it will ask you to select which "VRF" to use.

As long as you add the static route correctly, with the IP address and subnet as I stated, not larger, not smaller, it should work.
Try to ping the IP address to make sure you get a reply. If you don't get a reply, your static route setup is wrong.

If it didn't work, can you do a packet capture to make sure TM call server actually reply to your SIP packet?

The configuration for SIP server:
Primary Outbound Proxy Server: 10.225.15.220
Secondary Outbound Proxy Server: 10.225.17.220
Both port 5060.

A follow up explanation of why I do things like that for those who are into networking but don't quite get the whole static route thing.

On a high level, what we are doing is leak a very specific route from the VoIP VRF, to the global VRF of the ONU.
Then we perform the forwarding between global VRF of the ONU and global VRF of the router.

The exact route we are leaking is 10.225.0.0/16. I derive this from the IP address of the call server.
Most of the vendor call it global VRF. Mikrotik call it main VRF. They are the same.

You can refer to this website for the explanation since they have a static route example which I am using:
https://networklessons.com/cisco/ccie-routi...e-route-leaking.

You need to subscribe to read the whole thing but I feel reading the free section is already enough. If you are doing MPLS and BGP you probably already know the rest and don't need to subscribe.

Anybody who want to buy this, keep in mind:
1. The spec sheet is wrong. It uses a very rare 32-bit ARM CPU, same like hex refresh.
2. Read the block diagram. SPF and LAN1 are both dedicated port to the CPU. Meaning if you do switching on them, they will be processed by the CPU instead of switch chip. So for optimal performance, LAN2 - LAN5 should be used for switching while SPF + LAN1 should be used exclusively for routing. The same is true for LAN1 on hex refresh.
3. No L3 Hardware Offload at all.

Other than that, this thing looks good. The use case is niche but considering the price, it is great.
Example:
Your house isn't big.
Your internet speed is 1G or less but you want to use PON stick.
You also have one PoE out port for WiFi allowing optimal position.

I will go with hap ax3 unless your budget is constrained.
In fact I will actively discourage people from buying hap ax2.
The reason is the ax2 is rigged. Both uses the exact same SoC but I suspect due to the smaller chassis, there's thermal problem. Mikrotik limit the SoC frequency on ax2 so you are buying hardware that you cannot use to it's full potential.
Whether the discount justify the performance drop is a very personal preference.
Ax3 also gives you one 2.5g port and higher gain 5ghz antenna. You need to consider if those are thing you consider important or totally irrelevant. Ax3 also has a higher RouterOS license level but for home use that's irrelevant.
Ax2 has a higher gain 2.4ghz antenna. If your IoT is 2.4ghz then this might be a consideration.

To answer your goals.
Do you intent to use your hap ax wifi as well or you only want to stick to your mesh?
Things are not so straightforward when it comes to isolating devices.
Your wifi need to be able to segment those IoT device to a different network, via a different SSID to a different VLAN that Mikrotik can act against.
Easiest is to connect all you IoT to the Mikrotik wifi.
Continue to use your mesh for non-isolated devices.

When you say you want to isolate your wired TV box but I don't see it in your diagram. You must connect it directly to Mikrotik for it to be isolated. Isolation by IP address is not security as attacker can change IP. Plus your layer 2 network is still visible to attacker.

You can change your DHCP server setting and point to Pi-Hole.

If you want to restrict Android TV to only be able to communicate with your Pi, then you need to have the port connected to your Pi to act as different network as well. In the Interface List, configure all the ports accordingly so you can just use the list in firewall configuration. You get an advantage no layer 2 attack can hit your Pi, from any network.

I am going to tell you the truth, there's nothing specific to optimize for gaming. Just configure EIM-NAT, enable UPnP for your game to open port. That's all.
Traffic QoS are not effective in my opinion. They only work when your own network is congested. It won't help you when TM is congested. Doing QoS also effectively disable FastTrack so all your packet route slower. You also need QoS processing, which isn't free, all of them increases latency.
Lately, QoS only affect transmit. You cannot control in which order TM sends you traffic or how their HQoS prioritize your packet.

Mikrotik default firewall rules already did all that, for both IPv4 and IPv6.


Final note: TM only give you one IPv6 subnet. So you need to choose which network to have it. All other network cannot have IPv6 GUA. In this case, I think your PC and non-isolated wifi device will be on the same network, hence this is the one you use IPv6.

Configuration wise is another big topic. Either using VLAN, use routed port, do a hybrid. That's up to you

Most secure and reliable is just use routed port. It won't introduce network loop, it won't allow broadcast storm to propagate.Less overhead. No 4094 network limit.

The cons is you must buy layer 3 managed switch if you decide to use one later. Not only that, you have to consider which routing protocol to use. OSPF is a common one. The reason is many big player charge extra for switch to do BGP.
If you stay in Mikrotik ecosystem, this is irrelevant as Mikrotik do everything.

On high level those looks correct.
I highly recommend using Interface List because it will make your firewall rules easier to manage and less error prone. It also makes them easier to read, kind of self-documented.

If you use VLAN, you only need a L2 switch that can do VLAN. If you use routed port, you need a managed L3 switch. Either way a dumb switch won't work.
OK to add here... Since you are home user, you can just use static route instead of routing protocol as suggested from my previous post. I am sure you won't have more than 20 networks at home.

QoS only works if your own network upload is pegged or congested. It will otherwise not work. You will have better luck trying different VPN server till you get an uncongested circuit.