ISP connection across two data centres over DWDM

This was a typical smart hands task. The client had ordered a new ISP connection, it landed in one data centre, and the router it needed to reach sat in another.

They already had a DWDM link running over dark fibre between the two sites, so the path existed. The DWDM is theirs, with a dedicated internal team who look after it. What was needed was someone at each end to make the connections, and someone able to work out what was wrong if it did not come up.

Their technical team is remote, including the DWDM team. Nobody from the client was going to be in either building.

Work like this is complex for two reasons.

The first is that it happens in two places at once. An engineer in one building can only see half the link. Anything that goes wrong sits somewhere in a chain that runs from a patch panel in one data centre, down a dark fibre, through a DWDM system and into a router in another building. Testing it means both ends doing the same thing at the same time and comparing what they see.

The two sites belong to different colocation providers, which makes that harder than it sounds. Using each facility’s own remote hands desk would mean two separate suppliers, two ticket queues and two sets of engineers with no way of talking to each other while a test is running. Neither provider is set up to work in step with the other. Those teams also vary in what they are equipped for, and fibre testing is not always part of it.

The second is that the fault could be anywhere in that chain. The ISP owns the circuit. The router and the DWDM are the client’s, and the DWDM is looked after by a specialist team. Whoever is on site has to be the hands and eyes for all of it.

That is where a job like this escalates. The connection is made, the port does not come up, and the visit ends there. Resolving it then means further site visits at both ends, booked around two colocation providers and whichever engineer happens to be free, with email chains and coordination in between while each party explains why the problem is not theirs. A day of work turns into weeks.

The client shared all the information we needed up front. We put engineers in both buildings, made the connections, diagnosed the fault when the port stayed down, worked with the client’s DWDM team the same afternoon and stayed until the circuit was live.

Coordination was as much of the job as the cabling. One party held the circuit, another the router, another the DWDM, and the two buildings were run by different operators. Keeping all of them working on the same problem at the same time is what turned a fault that could have run for weeks into an afternoon.

Put a field engineer in each data centre on the same day. One at the site where the ISP circuit landed, one at the site with the router. This is our standard approach to anything spanning more than one building, and scheduling both for the same day is what made it possible to test the link properly rather than in two halves a week apart.

Put everyone in one conversation. Our engineers and the client’s technical contact were in the same Slack channel throughout. That is also standard for us: on jobs like this we get every party who might be needed into a single channel, third parties included, and where a shared channel is not possible we exchange phone numbers before the work starts so anyone can be reached immediately. Email is too slow for work where one end needs to know what the other end is seeing right now, and a shared channel leaves a record of what was tried, with photographs, that everyone can read afterwards.

Made the connections, then dealt with what happened next. Cabling was laid and the connections made at both ends. The router port did not come up.

Troubleshot the fibre path from both ends at once. This is fibre work rather than copper, and our engineers are trained for it, including the fibre hygiene that a surprising number of faults come down to. Working together across the two sites, they went through the path systematically, inspected and cleaned connectors, checked light levels and optics, and compared what each end was seeing. Everything on the physical side checked out, which left the DWDM as the remaining candidate.

Escalated the same day, with a conclusion rather than a question. Rather than closing the visit and handing back an unresolved fault, we raised it with the client’s DWDM team that afternoon. Because the physical path had already been eliminated, they could start on the DWDM itself instead of from nothing.

Acted as the DWDM team’s hands and eyes. DWDM is a telecommunication discipline and we do not claim it as ours, but our engineers understand how the system works, which is what made them useful. They could carry out what was asked at both ends of the link, describe accurately what they were seeing, and follow the reasoning. Nobody had to explain the system from scratch before the work could start. The channel in use was confirmed faulty.

Finished the job once the decision was made. With the fault confirmed, the client’s main contact decided to move the circuit to a different channel. We made the change, the circuit came up, and the labelling was updated to match so the next person to look at those panels sees what is actually in use.

The circuit was live the same day, on an alternative DWDM channel, after the original channel was confirmed faulty.

The physical work was a few hours. What decided the outcome was what happened when the port stayed down, and everything after that point was done by the same two engineers on the same day. The client did not attend either site, did not have to coordinate between buildings, and did not have to find a slot for a return visit.

Smart hands at two sites, with fibre troubleshooting and support to the client’s specialist teams, run by DACPROS.

If you need work carried out at more than one site at once, we can put engineers in both buildings on the same day and keep them talking to each other.