<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Replying the to the latest fully-quoted message instead of Ed's, but
Ed your observations are spot on.<br>
<br>
Rob, I think the concept of network ops is finished both in the
industry and for HamWAN. In the industry, we're working at such
enormous scales that you cannot possibly staff enough people to do
any of the ops tasks manually. Even if you did, the unavoidable
human failure rate would cripple your resulting system. In HamWAN,
we have the same problems as industry (albeit at a microscopic
scale), but additionally requiring staff to operate things is an
adoption hurdle. We don't have the incentive of wages to staff
these required job functions. Combine that with a general lack of
computer/network knowledge in the ham community and you're doomed,
even if you did manage to gather enough well-meaning people to
support you.<br>
<br>
This problem isn't unique to the Puget Sound Data Ring. Everyone
else trying to implement a HamWAN will face the same challenges, as
Ed correctly points out. We need to make the leap from phase 1 to
phase 2 (see Ed's email), because we've been successful enough
(yay!) to grow to such a scale that we're starting to fail at phase
1.<br>
<br>
HamWAN has so far delivered interfacing standards, and a bunch of
docs that educate people on suggestions (not standards) for how to
configure the non-standardized parts of your network. That's a good
starting point, but now that we know our standard ideas work
reasonably well, it's time to take on the additional task of making
them self-implementing in new HamWAN instances. This means a lot of
software development.<br>
<br>
And therein lies the problem. In this project we have maybe 2
people who can help write the software required. For us to
successfully make the leap from phase 1 to phase 2, we've got to
become attractive to people who write software. A team of 6-10
folks would give us a good chance at making the leap.<br>
<br>
I'm not sure how to do recruiting for this, but don't let that be
the seminal question of this email. I'd like to hear from people if
they agree with the direction shift I've proposed here.<br>
<br>
--Bart<br>
<br>
<br>
<div class="moz-cite-prefix">On 3/11/2016 10:25 AM, Sam Kuonen
wrote:<br>
</div>
<blockquote
cite="mid:CAG6u1mVU+jHM0bsM3LCdSemQ3-17M+VtHyghj4FCKGEqA34QPg@mail.gmail.com"
type="cite">
<p dir="ltr">I'll echo the time constraints. We're looking at core
infrastructure deployment for Georgia, USA and have a lot of
generalized interest in the project.</p>
<p dir="ltr">We're experiencing similar volunteer constraints and
have yet to begin full operations. I can only picture how
physical network operations are going to proceed and suffer once
those deployments start. <br>
</p>
<p dir="ltr">Regards,</p>
<p dir="ltr">Sam Kuonen, KK4UVL</p>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Mar 11, 2016, 12:29 PM Nigel Vander
Houwen <<a moz-do-not-send="true"
href="mailto:nigel@nigelvh.com">nigel@nigelvh.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Bart, Rob,</div>
<div><br>
</div>
<div>The biggest problem I see here is time resources. I
brought this up to Bart off list, but there’s a continuing
struggle to either have time to do the work yourself, or
get other people to do the work.</div>
<div><br>
</div>
<div>I deployed all of our monitoring and logging
infrastructure, and I can say as a fact it’s been a
struggle to get anyone to even do the basic work of adding
new devices to the existing monitoring system, even after
providing tutorials. This has gotten a bit better in very
recent history, but it remains an issue.</div>
<div><br>
</div>
<div>Automation is absolutely something we need to put more
work into. Ryan and I have already put a bunch of work
into this, which again, we have struggled to get folks to
pick up, use, and contribute to.</div>
<div><br>
</div>
<div>Modems breaking happens, and site access can be a
significant problem. The East Tiger-SnoDEM link that Bart
called out has been known down, but we can’t feasibly get
that replaced in the middle of winter. Hopefully soon that
can be taken care of.</div>
<div><br>
</div>
<div>We can try to treat this like a production network all
we want, but the reality is that we have effectively one
part time staff trying to do, as Rob put it, both the
Operations and Development work.</div>
<div><br>
</div>
<div>The reality is that this is a network with VERY limited
admin resources, which get split up to do various
important things, the 900MHz work included, but that
leaves even less available to do any day to day work. This
isn’t our full time job, we’re not paid, we all have lives
and families, we have VERY few people that actually
volunteer to do any of the work, so the reality is there’s
a lot we have a hard time getting to. Reality puts us much
closer to “best effort” than “production”, and until we
get more time/resources to do the work, it’s going to
continue to be a struggle.</div>
<div><br>
</div>
<div>If folks want to volunteer, I’d be happy to put them on
improvements in monitoring, automation, and fixing things
in the existing production network. </div>
</div>
<div style="word-wrap:break-word">
<div><br>
</div>
<div>Nigel</div>
</div>
<div style="word-wrap:break-word"><br>
<div>
<blockquote type="cite">
<div>On Mar 11, 2016, at 09:11, Rob Salsgiver <<a
moz-do-not-send="true" href="mailto:rob@nr3o.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rob@nr3o.com">rob@nr3o.com</a></a>> wrote:</div>
<br>
<div>
<div
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Bart,</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">You
touch on a few things that have been “niggling”
at the back of my mind for quite a while now –
most of them come down in one way or another to
overall reliability (of HamWAN) for EMCOMM,
which most know has been my main driver for
supporting the effort. <span> </span></span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">There’s
been a TON of great work done and quite frankly,
I’ve been amazed that HamWAN has gone as far and
fast as it has, particularly for a “ham”
effort. <span> </span></span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">At
the same time we’ve slowly been adding and
attracting the attention of various EMCOMM
organizations with the promise and potential of
redundant, reliable, resilient communications
when “the big one” hits. Obviously not
everything HamWAN is expected to survive a major
quake or other event, but even pockets of
reliable, high-speed communication are more than
what can be accomplished via voice relays.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">All
of which bring back to the current outage and
discussion. There have been several outages in
key places since we began. Last year SnoDEM was
all but stranded due to a Haystack modem failure
and other events at the same time. Now we have
a similar situation in a different place brought
on by multiple failures or weaknesses. In other
instances I’ve been told we’ve had outages via
misconfigured devices or other reasons. Even in
a perfect world, human error happens.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I
believe HamWAN would benefit from somewhat of a
shift in operating philosophy that would create
two separate departments or divisions –
operations and development. <span> </span></span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Operations
responsibilities</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Provide
day to day monitoring of network resources and
conditions</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Manage
(admin) of those portions of the network that
are designated as “in production”. This should
be the majority of the network.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Provide
communications and coordination of network
maintenance</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>4)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Maintain
an active inventory of all operational
(production) sites, site hardware, and site
access information.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>5)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Maintain
and manage all production site device
configurations and config change management.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>6)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Coordinate
implementation of new functionality introduced
by the Development department with appropriate
monitoring, end-user communication, etc</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>7)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Recommend
topics and technologies to be explored by the
Development team to enhance operational
stability and delivery of new features to the
network.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>8)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Document
technologies, methods, and tools selected for
use (and why) from an operational standpoint.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>9)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Maintain
an active inventory of spare hardware to support
all sites.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>10)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Establish
a plan to correct ALL key site failures within
XXXX days.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>11)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Coordinate
with Development to actively inject and test
network failures and redundancy capabilities.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>12)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Coordinate
with Development to enhance HamWAN’s ability to
operate in “pockets” when portions of the
network fail in an earthquake – i.e. – each
“island” stays operational with as many services
as possible</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Development
responsibilities</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Continued
exploration of new hardware, software, and
network management tools (Quagga vs BIRD, Metals
vs QRTs, etc)</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Conduct
experimentation with new hardware and software
on separate network resources where possible, or
in coordination with Operations on the larger
network (more on this below).</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Document
technologies, methods, and tools explored and
indicate pros/cons of each where possible.</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>4)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Continued
exploration, analysis, and documentation of
available antenna and shielding designs</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>5)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Exploration
of new antenna designs and/or other hardware?</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>6)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Exploration
of new frequencies and how they are affected by
terrain, vegetation, weather, etc</span></div>
<div style="margin:0in 0in 0.0001pt
0.5in;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>7)<span
style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times
New Roman'"> <span> </span></span></span></span><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">This
particular list can go on FOREVER</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The
distinction here is largely mental, but it’s
important. It is entirely probable to have the
same people in both groups, yet having the
separation is important if HamWAN wishes to be
taken seriously as a services provider to the
EMCOMM community. Any benefits from that would
also improve service for ALL HamWAN users.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Having
EMCOMM onboard is important. Not only does it
provide a needed service to them, but if
critical mass can be achieved it gives HamWAN
access to multiple sites in every city and
county. In turn though, HamWAN as a network
needs to be reliable in the “customer’s” eyes.
This means that infrastructure is managed with
uptime as the highest priority, experimentation
is managed to minimize adverse production
impacts, and equipment failures are identified
and corrected quickly.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">This
is admittedly a fair amount of work. Much of it
I suspect is already underway – maybe not just
quite in this format. Additional help will
definitely be useful. Everyone involved only
has so much time available, and they should be
able to focus on those items that are important
to them. I believe the above framework (or
something similar) begins to put some useful
structure in place that continues to shape
HamWAN from being the “wild west” of amateur and
network “geek” exploration into the reliable,
commercial grade, disaster resistant, amateur
platform it envisions to be - while still
allowing amateurs to push the limits of
technology like they are meant to.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If
the above (or something similar) is of interest
to the current directors and group as a whole,
we can easily create a similar worklist that
individuals on the sidelines can start picking
things they can help with to help bring about.</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Just
ideas. Not saying they’re perfect, but it’s a
start. Any other thoughts?</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Cheers,</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Rob
Salsgiver – NR3O</span></div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div>
<div style="border-style:solid none
none;border-top-color:rgb(225,225,225);border-top-width:1pt;padding:3pt
0in 0in">
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><b><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext">From:</span></b><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:windowtext"><span> </span>PSDR
[<a moz-do-not-send="true"
href="mailto:psdr-bounces@hamwan.org"
target="_blank">mailto:psdr-bounces@hamwan.org</a>]<span> </span><b>On
Behalf Of<span> </span></b>Bart Kus<br>
<b>Sent:</b><span> </span>Friday, March 11,
2016 12:56 AM<br>
<b>To:</b><span> </span><a
moz-do-not-send="true"
href="mailto:psdr@hamwan.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:psdr@hamwan.org">psdr@hamwan.org</a></a><br>
<b>Subject:</b><span> </span>Re: [HamWAN
PSDR] Service Impact Notice</span></div>
</div>
</div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"> </div>
<p class="MsoNormal" style="margin:0in 0in
12pt;font-size:12pt;font-family:'Times New
Roman',serif">Hmm that's not the whole story
though. If it were just the 1 router failure (in
reality a hypervisor failure), we'd be in a much
better position, but it's combined with 2 other
modem failures. We had the ETiger->SnoDEM
modem die over the winter, and it needs
replacement. That link has been down for a month
or more now. And most recently we're having the
Tukwila->Baldi modem lose connectivity
frequently. We've implemented an automatic
mitigation for that, but it still produces
sporadic short downtime windows of a few minutes.
I'd just like to move that modem to a NetMetal 5.
Our servers are also being affected by instability
in the Quagga routing software. We need to
replace this with a more stable alternative, like
BIRD. Lastly, the Baldi emergency uplink is only
configured to go to Westin and Corvallis, but not
Tukwila.<br>
<br>
We could have avoided DNS outages too, if the
anycast groups were populated with more of the
available servers. I believe lack of good
automation for server build-outs is causing the
deployment lag here.<br>
<br>
The network is designed to withstand failures,
even multiple failures, but we've got many broken
things right now that need fixing. After that
fixing, I would really love to see some folks get
behind improving our monitoring, deployment and
diagnostic automation. Networks like this won't
scale unless they're nearly completely automated
and simple to manage. I would not mind at all if
we even rolled back some features until we can get
them re-implemented in 100% automated ways.<br>
<br>
As important as all this is, I still think the
deep penetration project takes precedence, so I
can't drop that work in favor of this. Aside from
helping out on the simple break-fix stuff, I mean.<br>
<br>
--Bart<br>
<br>
</p>
<div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif">On 3/9/2016 8:23 PM, Ryan Elliott
Turner wrote:</div>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt"
type="cite">
<div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif">Thanks for the update, Nigel.</div>
</div>
<div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"> </div>
<div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times
New Roman',serif">On Wed, Mar 9, 2016 at
10:17 PM, Nigel Vander Houwen <<a
moz-do-not-send="true"
href="mailto:nigel@nigelvh.com"
style="color:purple;text-decoration:underline"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:nigel@nigelvh.com">nigel@nigelvh.com</a></a>>
wrote:</div>
<blockquote style="border-style:none none none
solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in
0in 0in
6pt;margin-left:4.8pt;margin-right:0in"
type="cite">
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times
New Roman',serif">Hello All,<br>
<br>
Just wanted to send out a quick notice
here. We’ve had a failure at our Seattle
edge router, which we’re still
investigating. In the meantime, our
Tukwila edge router is still providing
connectivity, but you may notice higher
latencies or issues reaching things. If
you find things you can’t reach, please
let me know, as we’d like to make sure the
redundancy is working, while we’re working
to resolve the issues we’re investigating
with the Seattle edge router.<br>
<br>
Nigel<br>
_______________________________________________<br>
PSDR mailing list<br>
<a moz-do-not-send="true"
href="mailto:PSDR@hamwan.org"
style="color:purple;text-decoration:underline"
target="_blank">PSDR@hamwan.org</a><br>
<a moz-do-not-send="true"
href="http://mail.hamwan.net/mailman/listinfo/psdr"
style="color:purple;text-decoration:underline" target="_blank">http://mail.hamwan.net/mailman/listinfo/psdr</a></div>
</blockquote>
</div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><br>
<br clear="all">
</div>
<div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times
New Roman',serif"> </div>
</div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif">--<span> </span></div>
<div>
<p
style="margin-right:0in;margin-left:0in;font-size:12pt;font-family:'Times
New Roman',serif"><span
style="font-family:Arial,sans-serif">Ryan
Turner</span></p>
</div>
</div>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"><br>
<br>
<br>
</div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">_______________________________________________</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">PSDR mailing list</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a moz-do-not-send="true" href="mailto:PSDR@hamwan.org" style="color:purple;text-decoration:underline" target="_blank">PSDR@hamwan.org</a></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a moz-do-not-send="true" href="http://mail.hamwan.net/mailman/listinfo/psdr" style="color:purple;text-decoration:underline" target="_blank">http://mail.hamwan.net/mailman/listinfo/psdr</a></pre>
</blockquote>
<div style="margin:0in 0in
0.0001pt;font-size:12pt;font-family:'Times New
Roman',serif"> </div>
</div>
<span
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">_______________________________________________</span><br
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
<span
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">PSDR
mailing list</span><br
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
<span
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important"><a
moz-do-not-send="true"
href="mailto:PSDR@hamwan.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:PSDR@hamwan.org">PSDR@hamwan.org</a></a></span><br
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
<span
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important"><a
moz-do-not-send="true"
href="http://mail.hamwan.net/mailman/listinfo/psdr"
target="_blank"><a class="moz-txt-link-freetext" href="http://mail.hamwan.net/mailman/listinfo/psdr">http://mail.hamwan.net/mailman/listinfo/psdr</a></a></span><br
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
</div>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
PSDR mailing list<br>
<a moz-do-not-send="true" href="mailto:PSDR@hamwan.org"
target="_blank">PSDR@hamwan.org</a><br>
<a moz-do-not-send="true"
href="http://mail.hamwan.net/mailman/listinfo/psdr"
rel="noreferrer" target="_blank">http://mail.hamwan.net/mailman/listinfo/psdr</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
PSDR mailing list
<a class="moz-txt-link-abbreviated" href="mailto:PSDR@hamwan.org">PSDR@hamwan.org</a>
<a class="moz-txt-link-freetext" href="http://mail.hamwan.net/mailman/listinfo/psdr">http://mail.hamwan.net/mailman/listinfo/psdr</a>
</pre>
</blockquote>
<br>
</body>
</html>