<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Moving PSDR@ to BCC so as to not continue spamming people as we
debug this.<br>
<br>
Ugh, the problem is not presenting itself today in the same way.
:( But here was the gist of it as of last night:<br>
<br>
<tt>21:17 <@EO_> [<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] /routing ospf lsa>
/ip route check 44.24.242.36</tt><tt><br>
</tt><tt>21:17 <@EO_> status: ok</tt><tt><br>
</tt><tt>21:17 <@EO_> interface: wlan1</tt><tt><br>
</tt><tt>21:17 <@EO_> nexthop: 44.24.242.39</tt><tt><br>
</tt><tt>21:17 <@EO_> [<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] /routing ospf
lsa> /ip route check 44.24.242.37</tt><tt><br>
</tt><tt>21:17 <@EO_> status: ok</tt><tt><br>
</tt><tt>21:17 <@EO_> interface: ether1</tt><tt><br>
</tt><tt>21:17 <@EO_> nexthop: 44.24.241.37</tt><tt><br>
</tt><br>
1) The modem being asked about the route decision is
"QueenAnne.Haystack", which is @ Haystack, RF-linked to QueenAnne
site and running OSPF with it.<br>
2) The destination IP of 44.24.242.36 is bound to
QueenAnne.CapitolPark, which is @ CapitolPark, RF-linked to
QueenAnne site and running OSPF with it.<br>
3) The destination IP of 44.24.242.37 is bound to
CapitolPark.QueenAnne, which is @ QueenAnne, RF-linked to
CapitolPark site and running OSPF with it.<br>
<br>
In other words, .36 and .37 are 2 sides of an RF link between
QueenAnne and CapitolPark. .36 sits on the CapitolPark side and .37
sits on the QueenAnne side. The shortest path to both IPs ought to
be via the Haystack->QueenAnne wlan1 RF hop, and then through the
LAN @ QueenAnne for .37, and through a 2nd RF hop for .36. That
means .36 ought to be MORE DISTANT than .37. Meanwhile, the routing
table lookup shows wlan1 being taken for .36 (the further
destination) and ether1 being taken for .37 (the closer
destination). The ether1 path is longer than the RF path, as it has
to traverse the Haystack LAN, then an RF link to K7NVH, then a VPN
tunnel to Seattle, then another RF link to QueenAnne, then the
QueenAnne LAN. This weird routing seems to have gone away today.<br>
<br>
The strange thing is, the route costs in OSPF persist at being
weird:<br>
<br>
<tt>65 44.24.242.36/32 intra-area
30 44.24.242.39 wlan1</tt><tt><br>
</tt><tt>66 44.24.242.37/32 intra-area
40 44.24.242.39 wlan1</tt><tt><br>
</tt><br>
Why would .37 be more costly here than .36? Here are the traces
today:<br>
<br>
<tt>[<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] /routing ospf route> /tool traceroute
44.24.242.36</tt><tt><br>
</tt><tt> # ADDRESS LOSS SENT LAST
AVG BEST WORST STD-DEV STATUS</tt><tt><br>
</tt><tt> 1 44.24.242.39 0% 2 31.7ms
33.7 31.7 35.7 2</tt><tt><br>
</tt><tt> 2 44.24.241.83 0% 2 4.8ms
11.2 4.8 17.5 6.4</tt><tt><br>
</tt><tt> 3 44.24.242.36 0% 2 11.5ms
9.7 7.8 11.5 1.9</tt><tt><br>
</tt><tt><br>
</tt><tt>[<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] /routing ospf route> /tool
traceroute 44.24.242.37</tt><tt><br>
</tt><tt> # ADDRESS LOSS SENT LAST
AVG BEST WORST STD-DEV STATUS</tt><tt><br>
</tt><tt> 1 44.24.242.39 0% 2 22.6ms
13.5 4.3 22.6 9.2</tt><tt><br>
</tt><tt> 2 44.24.242.37 0% 2 4.1ms
4 3.9 4.1 0.1</tt><tt><br>
</tt><br>
All interface costs along the path are "10", so I don't get this
OSPF calculation, even though the resulting route decision is
finally correct today. Here are the LSAs from CapitolPark.QueenAnne
and QueenAnne.CapitolPark, respectively, as seen by
QueenAnne.Haystack:<br>
<br>
<tt>[<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] > /routing ospf lsa print detail
where originator=44.24.241.83</tt><tt><br>
</tt><tt> instance=default area=backbone type=router id=44.24.241.83
originator=44.24.241.83 sequence-number=0x80000DB4 age=1225</tt><tt><br>
</tt><tt> checksum=0x6BD0 options="E" body=</tt><tt><br>
</tt><tt> flags=</tt><tt><br>
</tt><tt> link-type=Point-To-Point id=44.24.240.7 data=<b>44.24.242.37</b>
metric=10</tt><tt><br>
</tt><tt> link-type=Stub id=<b>44.24.242.36</b>
data=255.255.255.255 metric=10</tt><tt><br>
</tt><tt> link-type=Transit id=44.24.241.84
data=44.24.241.83 metric=10</tt><tt><br>
</tt><tt>[<a class="moz-txt-link-abbreviated" href="mailto:eo@QueenAnne.Haystack">eo@QueenAnne.Haystack</a>] > /routing ospf lsa print detail
where originator=44.24.240.7</tt><tt><br>
</tt><tt> instance=default area=backbone type=router id=44.24.240.7
originator=44.24.240.7 sequence-number=0x800045DD age=1240</tt><tt><br>
</tt><tt> checksum=0x754E options="E" body=</tt><tt><br>
</tt><tt> flags=</tt><tt><br>
</tt><tt> link-type=Point-To-Point id=44.24.241.83 data=<b>44.24.242.36</b>
metric=10</tt><tt><br>
</tt><tt> link-type=Stub id=<b>44.24.242.37</b>
data=255.255.255.255 metric=10</tt><tt><br>
</tt><tt> link-type=Transit id=44.24.240.7 data=44.24.240.7
metric=10</tt><tt><br>
</tt><tt><br>
</tt><tt> instance=default area=backbone type=network id=44.24.240.7
originator=44.24.240.7 sequence-number=0x800000C4 age=952</tt><tt><br>
</tt><tt> checksum=0xD1F1 options="E" body=</tt><tt><br>
</tt><tt> netmask=255.255.255.240</tt><tt><br>
</tt><tt> routerId=44.24.240.7</tt><tt><br>
</tt><tt> routerId=44.24.240.3</tt><tt><br>
</tt><tt> routerId=44.24.240.1</tt><tt><br>
</tt><tt> routerId=44.24.240.4</tt><tt><br>
</tt><tt> routerId=44.24.240.5</tt><tt><br>
</tt><tt> routerId=44.24.240.6</tt><tt><br>
</tt><br>
I have boldified the relevant IPs in each LSA. Can you make heads
or tails of why the weird costs are being computed?<br>
<br>
--Bart<br>
<br>
<br>
<div class="moz-cite-prefix">On 3/30/2018 9:59 AM, Dylan Ambauen
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEt0T_4hNUTYeD=3-MVgTatXNcSHx1PmvyR1JZzuT3FG1N9Uiw@mail.gmail.com">
<div dir="auto">Wow, Bart.
<div dir="auto"><br>
</div>
<div dir="auto">Let me look back at my ospf ptp configs. I'm no
expert, but I recall it was a process of elimination to find
the right settings. Are the ptp neighbors linking ospf in just
one direction?</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<br>
<br>
<div class="gmail_quote" dir="auto">
<div dir="ltr">On Thu, Mar 29, 2018, 23:17 Bart Kus <<a
href="mailto:me@bartk.us" moz-do-not-send="true">me@bartk.us</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> After hours of
debugging, I got this 20MHz single-polarity link moving up
to 55Mbit. A high resolution spectrum analysis on both
sides did indeed show a better frequency to use for the
link. Spectrum analysis captures here:<br>
<br>
<a class="m_-3474944101425623804moz-txt-link-freetext"
href="https://imgur.com/a/4H7GB" target="_blank"
rel="noreferrer" moz-do-not-send="true">https://imgur.com/a/4H7GB</a><br>
<br>
There was also an IP conflict between two modems @ Capitol
Park. The conflicting IP has been removed from
CapitolPark-S3.<br>
<br>
The QueenAnne modem used for the link is not one used
anywhere else on the network. It's a very low-end modem,
and as a result it was having CPU/RAM starvation issues
when running our regular diagnostic tools, which lead to
out-of-memory conditions and kernel crashes. A different
test methodology had to be used to verify the link speed
(testing through the modem, instead of to the modem).<br>
<br>
The modem that links QueenAnne with the Westin building
(on the QueenAnne side) had a mis-configured OSPF
router-id. This was fixed.<br>
<br>
I'm still seeing weird routing decisions being made by
OSPF. These are triggered by our point-to-point route
entries (<a href="http://44.24.242.0/24" target="_blank"
rel="noreferrer" moz-do-not-send="true">44.24.242.0/24</a>
space). More research needs to be done here, and perhaps
a re-write of how we define point-to-point interface
addresses. Any OSPF experts in the house?<br>
<br>
I also discovered R1.QueenAnne was still vulnerable to
hacking due to a mis-configuration of its control
software. It missed the updates that were sent out to the
whole network. This has been fixed now. R1.QueenAnne
also didn't have the diagnostic bandwidth-server setup
correctly. This was fixed.<br>
<br>
With the CapitolPark-QueenAnne link performing well now:<br>
<br>
<tt>[eo@CapitolPark-QueenAnne] /system resource> /tool
bandwidth-test 44.24.241.81 direction=both</tt><tt><br>
</tt><tt> status: running</tt><tt><br>
</tt><tt> duration: 57s</tt><tt><br>
</tt><tt> tx-current: 28.0Mbps</tt><tt><br>
</tt><tt> tx-10-second-average: 28.4Mbps</tt><tt><br>
</tt><tt> tx-total-average: 27.5Mbps</tt><tt><br>
</tt><tt> rx-current: 27.6Mbps</tt><tt><br>
</tt><tt> rx-10-second-average: 28.0Mbps</tt><tt><br>
</tt><tt> rx-total-average: 27.0Mbps</tt><tt><br>
</tt><tt> lost-packets: 288</tt><tt><br>
</tt><tt> random-data: no</tt><tt><br>
</tt><tt> direction: both</tt><tt><br>
</tt><tt> tx-size: 1500</tt><tt><br>
</tt><tt> rx-size: 1500</tt><tt><br>
</tt><tt><br>
</tt>its OSPF config has been reset to a normal preference
level, so that packets no longer try to avoid that link as
they are routed through the network. This link can be
sped up by upgrading to a dual-polarity modem @
CapitolPark.<br>
<br>
While testing if the OSPF hop cost was being calculated
correctly in the Beacon-Haystack-QueenAnne RF link (they
both connect to the same dish @ Haystack), I discovered a
mis-config on the Haystack.Beacon modem (bad LAN IP
binding) which was preventing it from bringing up OSPF on
its LAN interface. This was fixed and that modem should
act like an actual router now, moving traffic.<br>
<br>
During the same Beacon testing, I was reminded that our
Baldi-Beacon RF link <a
href="https://monitoring.hamwan.net/cacti/graph.php?action=view&local_graph_id=919&rra_id=all"
target="_blank" rel="noreferrer" moz-do-not-send="true">sucks</a>.
It was optimized for speed to Tukwila, which is now gone,
so a trip needs to be scheduled to Baldi to rotate the
dish a few degrees north and get that link going strong.<br>
<br>
I normally wouldn't send this kind of verbose email to
psdr@, but I hope it's illuminating as to the type and
extent of work required to keep this network running well.<br>
<br>
--Bart<br>
<br>
</div>
_______________________________________________<br>
PSDR mailing list<br>
<a href="mailto:PSDR@hamwan.org" target="_blank"
rel="noreferrer" moz-do-not-send="true">PSDR@hamwan.org</a><br>
<a href="http://mail.hamwan.net/mailman/listinfo/psdr"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://mail.hamwan.net/mailman/listinfo/psdr<br>
</a><br>
</blockquote>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
PSDR mailing list
<a class="moz-txt-link-abbreviated" href="mailto:PSDR@hamwan.org">PSDR@hamwan.org</a>
<a class="moz-txt-link-freetext" href="http://mail.hamwan.net/mailman/listinfo/psdr">http://mail.hamwan.net/mailman/listinfo/psdr</a>
</pre>
</blockquote>
<br>
</body>
</html>