Online S.A.S. - AS 12876 - joined the RING today. Online.net is one the biggest hosting company in France created in 1999, we offer a range of services including domain names, web hosting, dedicated servers and hosting in our own datacenters.
Users can connect to onlinesas01.ring.nlnog.net, which is located in France.
ATW Internet Kft. - AS 41075 - joined the RING today. ATW provides all kinds of internet-related services from domain registration to building and letting complete data centers, and almost everything in between.
Users can connect to atw01.ring.nlnog.net, which is located in Hungary.
A new partial outage detector dubbed RING SQA is available to all RING participants. The purpose of the tool is to detect outages as fast as possible that only affect a subset of all internet destinations.
RING SQA pings all other nodes (v4 + v6) every 30 seconds to derive a baseline, this baseline is compared to the last 3 minutes of measurements. If the median of the baseline is tripped for three consecutives minutes, an alarm is raised.
When an alarm is raised, three MTRs are immediately launched towards destinations that previously were reachable, but suddenly not anymore. The purpose of these traces is to provide an investigation starting point for your NOC.
All in all super fast outage detection. All participants are invited to use this system! Gratis! :-)
One can simply configure where alerts should be emailed by changing the /etc/ring-sqa/alarm.conf file on your own RING node(s) to something like this (do keep in mind the indenting!):
job@ringnode01.ring.nlnog.net:~$ sudo cat /etc/ring-sqa/alarm.conf
---
email:
to: noc@yourcompany.com
from: sqa-alert@ your_ring_node .ring.nlnog.net
prefix: 'RING ALERT '
irc:
host: 1.2.3.4
port: 5502
password: derp
channel: ! '#noc'
Afterwards restart the ring-sqa daemons to load the new config:
job@ringnode01:~$ sudo restart ring-sqa4
ring-sqa4 start/running, process 18715
job@ringnode01:~$ sudo restart ring-sqa6
ring-sqa6 start/running, process 18727
Et voila! After 30 minutes the machine will stand guard over your network. RING participants with multiple hubs or datacenters will benefit from spinning up more nodes, as monitoring is from each RING nodes individual perspective.
I’d like to extend a HUGE thank you to Saky Ytti who wrote RING SQA just last weekend. Please send him beer, chocolate and flowers.
Below I’ve included an example outage alert.
------------------ Example RING SQA Message ---------------------------
From: sqa@xing02.ring.nlnog.net
To: noc@
Subject: RING ALERT raising ipv4 alarm - 16 new nodes down
Body:
Regarding: xing02.ring.nlnog.net ipv4
This is an automated alert from the distributed partial outage
monitoring system "RING SQA".
At 2014-07-27 10:18:05 UTC the following measurements were analysed
as indicating that there is a high probability your NLNOG RING node
cannot reach the entire internet. Possible causes could be an outage
in your upstream's or peer's network.
The following nodes previously were reachable, but became unreachable
over the course of the last 3 minutes:
- itps01.ring.nlnog.net 128.65.97.93 AS42010 GB
- fullsave01.ring.nlnog.net 141.0.202.201 AS39405 FR
- globalaxs01.ring.nlnog.net 176.10.80.10 AS 9009 GB
- kwaoo01.ring.nlnog.net 178.250.209.33 AS24904 CH
- suretec01.ring.nlnog.net 185.8.92.17 AS199659 GB
- swisscom01.ring.nlnog.net 193.247.170.254 AS 3303 CH
- claranet01.ring.nlnog.net 195.157.9.4 AS 8426 GB
- claranet04.ring.nlnog.net 195.22.19.34 AS 8426 PT
- dcsone01.ring.nlnog.net 203.123.48.14 AS37989 SG
- trueinternet01.ring.nlnog.net 203.144.167.57 AS 7470 TH
- jump01.ring.nlnog.net 212.13.217.117 AS 8943 GB
- lchost01.ring.nlnog.net 213.230.217.125 AS25098 GB
- suomi01.ring.nlnog.net 217.119.42.194 AS16302 FI
- melbourne01.ring.nlnog.net 37.128.187.253 AS39451 GB
- netability01.ring.nlnog.net 46.182.9.20 AS 1197 IE
- viatel02.ring.nlnog.net 46.183.108.2 AS31122 FR
- claranet06.ring.nlnog.net 92.54.7.29 AS 8426 ES
As a debug starting point 3 traceroutes were launched right after
detecting the event, they might assist in pinpointing what broke:
trueinternet01.ring.nlnog.net AS 7470 (TH)
mtr -i0.5 -c5 -r -w -n 203.144.167.57
1.|-- 109.233.156.241 0.0% 6 0.5 0.5 0.5 0.6 0.0
2.|-- 109.233.156.1 0.0% 5 0.8 0.9 0.8 1.1 0.1
3.|-- 109.233.156.2 0.0% 5 0.8 0.8 0.8 0.9 0.0
4.|-- 64.209.88.33 0.0% 5 0.9 1.0 0.9 1.5 0.3
5.|-- 159.63.23.198 60.0% 5 265.1 264.9 264.7 265.1 0.3
6.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
7.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
8.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
9.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
10.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
11.|-- 203.144.144.30 80.0% 5 297.4 297.4 297.4 297.4 0.0
12.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
fullsave01.ring.nlnog.net AS39405 (FR)
mtr -i0.5 -c5 -r -w -n 141.0.202.201
1.|-- 109.233.156.241 0.0% 6 0.5 0.5 0.5 0.5 0.0
2.|-- 109.233.156.1 0.0% 5 0.8 3.2 0.8 12.2 5.0
3.|-- 109.233.156.2 0.0% 5 0.8 0.9 0.8 1.0 0.1
4.|-- 109.233.156.37 0.0% 5 1.0 1.0 0.9 1.5 0.3
5.|-- 149.11.106.1 0.0% 5 1.1 1.4 1.1 1.7 0.2
6.|-- 130.117.3.137 0.0% 5 1.5 1.7 1.5 1.8 0.2
7.|-- 154.54.62.77 0.0% 5 11.4 11.7 11.3 13.0 0.7
8.|-- 154.54.75.154 0.0% 5 201.0 166.9 66.9 323.0 101.5
9.|-- 154.54.56.214 0.0% 5 23.0 23.0 22.8 23.0 0.1
10.|-- 149.11.58.62 80.0% 5 26.4 26.4 26.4 26.4 0.0
11.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
12.|-- 141.0.202.201 80.0% 5 25.0 25.0 25.0 25.0 0.0
globalaxs01.ring.nlnog.net AS 9009 (GB)
mtr -i0.5 -c5 -r -w -n 176.10.80.10
1.|-- 109.233.156.241 0.0% 6 0.4 0.5 0.4 0.5 0.0
2.|-- 109.233.156.1 0.0% 5 0.9 1.8 0.7 5.3 1.9
3.|-- 81.201.115.41 0.0% 5 0.9 0.9 0.8 1.0 0.1
4.|-- 62.209.32.18 40.0% 5 1.3 1.2 1.2 1.3 0.1
5.|-- 80.81.192.165 0.0% 5 1.3 9.3 1.2 41.5 18.0
6.|-- 193.27.64.245 60.0% 5 191.9 108.1 24.3 191.9 118.5
7.|-- 193.27.64.66 80.0% 5 43.6 43.6 43.6 43.6 0.0
8.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
9.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
10.|-- 176.10.80.2 80.0% 5 26.1 26.1 26.1 26.1 0.0
11.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
13.|-- ??? 100.0 3 0.0 0.0 0.0 0.0 0.0
14.|-- ??? 100.0 2 0.0 0.0 0.0 0.0 0.0
15.|-- 176.10.80.10 0.0% 1 24.3 24.3 24.3 24.3 0.0
An alarm is raised under the following conditions: every 30 seconds
your node pings all other nodes. The amount of nodes that cannot be
reached is stored in a circular buffer, with each element representing
a minute of measurements. In the event that the last three minutes are
1.2 above the median of the previous 27 measurement slots, a partial
outage is assumed. The ring buffer's output is as following:
29 min ago 41 measurements failed (baseline)
28 min ago 41 measurements failed (baseline)
27 min ago 41 measurements failed (baseline)
26 min ago 42 measurements failed (baseline)
25 min ago 41 measurements failed (baseline)
24 min ago 41 measurements failed (baseline)
23 min ago 41 measurements failed (baseline)
22 min ago 41 measurements failed (baseline)
21 min ago 41 measurements failed (baseline)
20 min ago 41 measurements failed (baseline)
19 min ago 41 measurements failed (baseline)
18 min ago 41 measurements failed (baseline)
17 min ago 41 measurements failed (baseline)
16 min ago 41 measurements failed (baseline)
15 min ago 41 measurements failed (baseline)
14 min ago 41 measurements failed (baseline)
13 min ago 41 measurements failed (baseline)
12 min ago 41 measurements failed (baseline)
11 min ago 41 measurements failed (baseline)
10 min ago 41 measurements failed (baseline)
9 min ago 41 measurements failed (baseline)
8 min ago 41 measurements failed (baseline)
7 min ago 41 measurements failed (baseline)
6 min ago 41 measurements failed (baseline)
5 min ago 41 measurements failed (baseline)
4 min ago 41 measurements failed (baseline)
3 min ago 45 measurements failed (baseline)
2 min ago 66 measurements failed (raised alarm)
1 min ago 65 measurements failed (raised alarm)
0 min ago 65 measurements failed (raised alarm)
Kind regards,
NLNOG RING
Vibe Communications LTD - AS 45177 - joined the RING today. We’re an ISP in New Zealand, one of 5 carriers of transit out of the country. We provide mostly wholesale services to other ISP’s while also servicing our corporate and SME customer base.
Users can connect to vibe01.ring.nlnog.net, which is located in New Zealand.
Blizoo Media and Broadband - AS 13124 - joined the RING today. Blizoo Media and Broadband EAD is the biggest cable operator in Bulgaria and Makedonia with our own DWDM network covering both countries. We offer cable TV, Internet and Voice services to residential customers and telecommunication services for voice and data transfer to corporate clients.
Users can connect to blizoo01.ring.nlnog.net, which is located in Bulgaria.
Networkoperations - AS 30830 - joined the RING today. Networkoperations is a hosting provider specialized in custom hosted services for medium and large businesses.
Users can connect to networkoperations01.ring.nlnog.net, which is located in Netherlands.
Cloud.nl - AS 60950 - joined the RING today. Cloud.nl is a company which provides high availability solutions for high traffic websites. We can host the website on our own virtualisation platform or we manage the website running in the customers own network.
Users can connect to cloudnl01.ring.nlnog.net, which is located in Netherlands.
Suretec Systems Ltd. T/A SureVoIP - AS 199659 - joined the RING today. SureVoIP (Suretec Systems Ltd.) is an Ofcom registered Internet Telephony Service Provider supplying Business hosted VoIP solutions, Business SIP trunks, Business VoIP Inbound numbers, International SIP numbers and other related Business VoIP and Internet connectivity products and services in the UK.
Users can connect to suretec01.ring.nlnog.net, which is located in United Kingdom.
MK Netzdienste GmbH & Co. KG - AS 25394 - joined the RING today. MK Netzdienste is an ISP located in Germany providing normal Internet-Access, MPLS-based VPN’s, Cloud-Computing, Serverhousing, Webservices and Telephony-Services
Users can connect to mknetzdienste01.ring.nlnog.net, which is located in Germany.
noris network AG - AS 12337 - joined the RING today. Company description: noris network AG provides managed IT and IT outsourcing services in the areas Internet Services Providing (ISP), Virtual Private Network (VPN), Managed Security, VoIP, Application Hosting, Terminal Services, eCommerce Applications and Storage/Backup.
These services are provided on top of a high performance IP infrastructure with our own backbone and multiple high security datacenters.
Our brand datacenter.de provides standardized premium datacenter products.
Users can connect to noris01.ring.nlnog.net, which is located in Germany.