• Please review our updated Terms and Rules here

DEC-20 Day!

bqt

Veteran Member
Joined
Sep 15, 2019
Messages
506
Location
Zurich, CH
I'd be surprised, because I had 3 8650s at one time and did the breaker panels / input power for them. I don't recall any warning in the site prep or installation guides about ensuring correct phasing. Plus, there is an airflow sensor that trips the entire system offline if there is insufficient airflow. Simply trying to slide the reusable filter out of an 86x0 will trip the airflow sensor. Trivia - the 86x0 doesn't need that ridiculously expensive Russel Stoll 60A watertight connector. I ran our first one happily on am adapter cable I made to power it fom the L21-30R receptacle we used for the 785 it replaced.

Oh, believe me, it do use the three phases that way. I should say that I can't swear US models might have been different, but I know about the one some people I know wired up backwards. And yes, there is an airflow sensor that trips if you wire it wrong. They were also creative enough to wire around it, since they didn't realize the fans were running the wrong way. I can't remember if they even managed to cause some permanent damage to the machine. I could ask if they remember. It was a little more than 20 years ago now...

The fans in the 86x0 machines are pretty massive. It's a veritable storm behind the machine. But I suspect you already knew that. :)

And since Update still have it's 8650, I can even check that one as soon as I am in Sweden next time.

The one we have had an oversized power connection, but nothing close to what you describe. But it had a 32A 3phase connector of the standard style in europe. But we put an adaptor on to a 16A version, which works just fine. Rumours had it that they put that connector on to be able to just plug it in at the same installation as IBM machines they hoped to replace, which needed the 32A.
But I have no idea if that is true or not.

Our DEC-2060s had ridicoluous 63A 3phase connectors, though. But I'm not sure they usually were delivered with that or not... Can't believe they really needed something like that.

Just searched through some documentation on the 86x0. Here is the best I could find on the fans:
https://vt100.net/mirror/antonio/86xv1mg3.pdf
See page 18-25 (page 445). There you see mentioned that after replacing the blower motor, one should check the rotational direction, and if it is wrong, the harness connection should be checked for reversed leads. (Block is on previous page.)
If you search through some other parts, you'll find mentioned on the power distribution, that 4 wires goes to the blower motors. 3 phases plus neutral.

So it appears to be the same on US models. Different blower motors for 60Hz and 50Hz as well.

Oh. And I forgot: removal of the filter don't trip the airflow sensor. There is a separate switch detecting if the filter is in place, and power is immediately removed if the filter is not in place. But it's a separate switch to the airflow detection. It's also documented elsewhere in that manual.

I just love the 86x0 machines. Such much nice engineering in them. Unfortunately our 8650 only have 60Mb of memory. We did try to ship the cards from you Terry, a long time ago, but they disappeared somewhere along the way in shipping. No clue ever where they ended up. But still thanks for that effort.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
The fans in the 86x0 machines are pretty massive. It's a veritable storm behind the machine. But I suspect you already knew that. :)
There is no truth to the rumor that the machine is so heavy just to keep the machine from hovering above the floor due to the fans :p

The one we have had an oversized power connection, but nothing close to what you describe. But it had a 32A 3phase connector of the standard style in europe. But we put an adaptor on to a 16A version, which works just fine. Rumours had it that they put that connector on to be able to just plug it in at the same installation as IBM machines they hoped to replace, which needed the 32A. But I have no idea if that is true or not.
Mine were perfectly happy on 30A 120/208 3 phase.

The IBM plug story is absolutely true. This was during the era when DEC hired hundreds (thousands?) of IBM sales people to try to take some of IBM's market share. It was a typical later era DEC disaster, as those sales people had no idea what DEC was or meant, and they probably sold more gear for IBM after being hired by DEC, simply by being more familiar with it.

The only stuff I've ever seen that used those $1000+ Russel Stoll connectors is "big iron" IBM gear and smaller ones at marinas because they are watertight.

I just love the 86x0 machines. Such much nice engineering in them. Unfortunately our 8650 only have 60Mb of memory. We did try to ship the cards from you Terry, a long time ago, but they disappeared somewhere along the way in shipping. No clue ever where they ended up. But still thanks for that effort.
They are beautifully engineered, but were from DEC's LCG - the "perpetually too little, too late" group that also brought you the VAX 9000 and failed with Jupiter. The 8650 had the performance originally intended for the 8600 (an up-to-rev 8600 requires only 2 processor cards changed and new microcode to be an 8650). The "mid-life kicker" would have been the 8670, but by that time smaller VAXen were faster and ECL macrocells were an evolutionary dead end (a type of mistake LCG would make again with the Trilogy technology in the 9000).

Boards from SPC have ended up in strange places. When I was fixing the LSSM's 750 (which had a bad inner-layer etch on the backplane) we also suspected a bad Ethernet controller (it wasn't - we'd all forgotten the SQE switch on Allied Telesis AUI-to-BNC transceivers) and they brought out a DELUA from spares which I was amused to find had "11D" (as in spc11d.spc.edu) written on it in my writing. I have no idea what complex path got it to the LSSM.

Which reminds me - I have to get back there post-Covid to do some more work on their 780. I got it from dying at the LSI ODT prompt to getting to the point where "B DU0" actually initialized the UDA and selected the port on the RA90, but hung. There were around a dozen things wrong with it in between. At one point after I fixed something simply by glaring at it, Dave asked me what I did and I replied "They're afraid of me". I think he believed me, as he hasn't done anything further on that system since - he's waiting for me to come back.
 

bqt

Veteran Member
Joined
Sep 15, 2019
Messages
506
Location
Zurich, CH
There is no truth to the rumor that the machine is so heavy just to keep the machine from hovering above the floor due to the fans :p
:)
That's the other thing I can say about that machine. It's weight is almost ridicoluous. Update actually have two. But getting them was an interesting exercise... Thank god we have a proper computer hall with no doorsteps all the way from the outside in to the hall. The effort we had getting one of them out from where it was donated required getting over one door step, and we broke a lot of material getting it across...

Mine were perfectly happy on 30A 120/208 3 phase.
Yeah. Makes sense. We're running at 430V 3phase. Hence just 16A per phase...
The IBM plug story is absolutely true. This was during the era when DEC hired hundreds (thousands?) of IBM sales people to try to take some of IBM's market share. It was a typical later era DEC disaster, as those sales people had no idea what DEC was or meant, and they probably sold more gear for IBM after being hired by DEC, simply by being more familiar with it.

The only stuff I've ever seen that used those $1000+ Russel Stoll connectors is "big iron" IBM gear and smaller ones at marinas because they are watertight.
I should check those connectors out. Not familiar with them. Being european and all...
They are beautifully engineered, but were from DEC's LCG - the "perpetually too little, too late" group that also brought you the VAX 9000 and failed with Jupiter. The 8650 had the performance originally intended for the 8600 (an up-to-rev 8600 requires only 2 processor cards changed and new microcode to be an 8650). The "mid-life kicker" would have been the 8670, but by that time smaller VAXen were faster and ECL macrocells were an evolutionary dead end (a type of mistake LCG would make again with the Trilogy technology in the 9000).
Well, then 8600 and 8650 were quite successful anyway. But yes, they were late, and performance wasn't where it was planned at launch.

The 9000 was a much bigger failure.
Boards from SPC have ended up in strange places. When I was fixing the LSSM's 750 (which had a bad inner-layer etch on the backplane) we also suspected a bad Ethernet controller (it wasn't - we'd all forgotten the SQE switch on Allied Telesis AUI-to-BNC transceivers) and they brought out a DELUA from spares which I was amused to find had "11D" (as in spc11d.spc.edu) written on it in my writing. I have no idea what complex path got it to the LSSM.

Which reminds me - I have to get back there post-Covid to do some more work on their 780. I got it from dying at the LSI ODT prompt to getting to the point where "B DU0" actually initialized the UDA and selected the port on the RA90, but hung. There were around a dozen things wrong with it in between. At one point after I fixed something simply by glaring at it, Dave asked me what I did and I replied "They're afraid of me". I think he believed me, as he hasn't done anything further on that system since - he's waiting for me to come back.

Ah. I wish I could join in that. I would have so much fun... Things have a tendency to work where I'm involved as well. I'm probably just too stubborn to accept when things don't work. :)

But it's sometimes been interesting experiences... Both core memory boxes for the 11/70, cache memory, as well as the CPU timing modules have been issues at one time or another. (We do still have one MJ11 around, for nostalgic reasons, even if not installed). One 8650 all connected, and occasionally turned on, for fun. Unfortunately, our current cooling can't really deal with it running all the time. Enough work with just the 11/70...
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
It's weight is almost ridicoluous. Update actually have two. But getting them was an interesting exercise... Thank god we have a proper computer hall with no doorsteps all the way from the outside in to the hall. The effort we had getting one of them out from where it was donated required getting over one door step, and we broke a lot of material getting it across...
A lot of careful planning went into being able to get this stuff out of the old SPC building (McDermott basement) and into 121 Glenwood, a converted apartment building. The alleyway into the basement was just wide enough (brick on both sides) to roll the systems in the long way. Then there were double doors into the computer room lobby in the basement. But if you didn't remember to send the system down the alley pivoting wheels first, you had to get the brute squad to shove 1200+ pounds of computer across concrete 90 degrees from the axis of the wheels. There's actually a funny story about our 3138 (370/138) CPU getting away from the movers and rolling down Glenwood, across a busy avenue, and into a cemetary where it took a chunk out of the corner of a mausoleum and fell over.

I should check those connectors out. Not familiar with them. Being european and all...
According to IBM Publication GC22-7004-14, the 3040 motor-generator set uses the same RS connector for all variants:
Code:
Voltage*                                    200     208     220     230     235     380    408
Ampacity**                                   50      60      50      60      50      30     30

Here is the product page for the RS7428 connector. It can be yours for only $1160 (I bet you thought I was kidding when I said "$1000+ connector").

The 9000 was a much bigger failure.
The 86x0 had the advantage of being mostly done by the time it became obvious it wasn't going to be the knockout punch it was intended to be. The 8600 (4 VUPS) was released October 1984 and the 8650 (and upgrade kits) (7 VUPS, so the 8600 was a pretty big miss) in December 1985. So it took them over a whole year to meet their planned performance target. The 8800 family first shipped in January, 1986, only a month after the 8650. An 8800 was 12 VUPS with both CPUs, 6 VUPS in uniprocessor mode/8700 skin.

Meaningful shipment of the 9000 series didn't start until early 1991, by which time NVAX machines were performance competitive at a fraction of the price (VAX 9000-110, 40 VUPS, $920,000) vs. VAX 4000-700A, 1993, also 40 VUPS, $85,340). So less than 1/10th the price, even before adding maintenance, power and cooling costs.

Ah. I wish I could join in that. I would have so much fun... Things have a tendency to work where I'm involved as well. I'm probably just too stubborn to accept when things don't work. :)
If you ever get to the US East Coast, drop me a note first and we'll go out to the LSSM outside Pittsburgh. I can tell you from firsthand experience that Dave and the gang do a wonderful "We're not worthy!" impression (from "Wayne's World"). :D

But it's sometimes been interesting experiences... Both core memory boxes for the 11/70, cache memory, as well as the CPU timing modules have been issues at one time or another. (We do still have one MJ11 around, for nostalgic reasons, even if not installed). One 8650 all connected, and occasionally turned on, for fun. Unfortunately, our current cooling can't really deal with it running all the time. Enough work with just the 11/70...
After doing a load of work on the SPC 11/70's they were rock sold (after being donated because they were unreliable). I changed all of the fans to Torin TA450s which sounded wonderful when the machine was powered up, unlike the random screeching and groaning of the old fans. There were a bunch of ECOs that needed doing. The backplane didn't get reliable until around CS rev J (when they went to twisted pairs for important signals) and the last revision was rev T, I think. Adjusting the power supplies (and converting their burned-out bulbs to LEDs) also helped reliability. The flat ribbon cables (front panel, Massbus to bulkhead, and memory) were always problem points with the insulation being needle-pricked by IC legs. BC05, I think.
 

bqt

Veteran Member
Joined
Sep 15, 2019
Messages
506
Location
Zurich, CH
According to IBM Publication GC22-7004-14, the 3040 motor-generator set uses the same RS connector for all variants:
Code:
Voltage*                                    200     208     220     230     235     380    408
Ampacity**                                   50      60      50      60      50      30     30

Here is the product page for the RS7428 connector. It can be yours for only $1160 (I bet you thought I was kidding when I said "$1000+ connector").
Interesting. I'm not sure how spread the style of connectors we use in Europe is, but they are definitely cheaper. I checked around, and 400V 3phase, up to 125A, IP67 (so, water proof), cost about 300 euros.
Those are standard red connectors with 5 pins.
If you ever get to the US East Coast, drop me a note first and we'll go out to the LSSM outside Pittsburgh. I can tell you from firsthand experience that Dave and the gang do a wonderful "We're not worthy!" impression (from "Wayne's World"). :D
Unfortunately, my current job don't really lead me to travels to the civilized world. So at the moment, it might take a while... But I'll keep it in mind.
After doing a load of work on the SPC 11/70's they were rock sold (after being donated because they were unreliable). I changed all of the fans to Torin TA450s which sounded wonderful when the machine was powered up, unlike the random screeching and groaning of the old fans. There were a bunch of ECOs that needed doing. The backplane didn't get reliable until around CS rev J (when they went to twisted pairs for important signals) and the last revision was rev T, I think. Adjusting the power supplies (and converting their burned-out bulbs to LEDs) also helped reliability. The flat ribbon cables (front panel, Massbus to bulkhead, and memory) were always problem points with the insulation being needle-pricked by IC legs. BC05, I think.
I am sometimes amazed/confused by places that have an 11/70, and seem to have such problems keeping it running. Admittedly, we do have lots of spares at Update (we actually have three 11/70 machines), but if needed, I'm pretty confident I could get them all running perfectly fine. As it is, we have one running 24/7, and the other two are in storage, and not fully populated. Lots of cards in boxes... We probably have enough cards for at least yet one machine.
The biggest issue in general would be the H744 regulators. And I am not good at analog electronics. That's where I fail completely.
We have lots of failed regulators. But there is a guy who took a look at some of them, and fixed four for us.

Fans are a bit of an issue in that when checking around, I sometimes spot a fan not working. But I have never had a machine have any issues, even when more than half the fans were not running.

By the way, I do not think I've seen a good summary of all the ECOs for the 11/70. Would have such a list? I would like to check at what ECOs our machines are.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
I am sometimes amazed/confused by places that have an 11/70, and seem to have such problems keeping it running. Admittedly, we do have lots of spares at Update (we actually have three 11/70 machines), but if needed, I'm pretty confident I could get them all running perfectly fine. As it is, we have one running 24/7, and the other two are in storage, and not fully populated. Lots of cards in boxes... We probably have enough cards for at least yet one machine.
The biggest issue in general would be the H744 regulators. And I am not good at analog electronics. That's where I fail completely.
We have lots of failed regulators. But there is a guy who took a look at some of them, and fixed four for us.
The last remaining 11/70s were performing mission-critical services. Some of the last ones I knew of were at Hydro Quebec (including an 11/74!), various Bell Telephone installations (replaced by WeCo 3B20 systems and later by Sun SPARC systems) and the Los Angeles E911 dispatch system.

Fans are a bit of an issue in that when checking around, I sometimes spot a fan not working. But I have never had a machine have any issues, even when more than half the fans were not running.
It is more of an aesthetic thing. There's nothing like powering on an 11/70 with a bunch of factory-fresh fans. It isn't a job for thr faint of heart, though - there are a LOT of fans (32), some of them in very inconvenient places, IIRC.

By the way, I do not think I've seen a good summary of all the ECOs for the 11/70. Would have such a list? I would like to check at what ECOs our machines are.
You might find DECUServe HARDWARE_HELP 238.x helpful. It is abstracted in http://www.people.vcu.edu/~agnew/MVAX/9412_DECUSERVE_JNL.HTML starting on Page 31. One of the things in there is the last known revision list for all 11/70 components. As I noted there, "This information is especially important as the DEC FCO documentation (DEC-O-LOG, MDS) is *incorrect* and *incomplete* for the 11/70." so other than verifying revisions, I don't know if the sites with archived DEC-O-LOG fiche have the correct info to actually perform the ECO/FCO.

It is hard to believe that I wrote that stuff over 32 years ago. :eek: There is also a bunch of other useful stuff from me (and others) at that link. Here's the table of contents from that issue:

Code:
                                   CONTENTS

                From the Editor's Keyboard . . . . . . . . . . . . . 1
                Table of Contents  . . . . . . . . . . . . . . . . . 2
                BA23 Fire Hazard . . . . . . . . . . . . . . . . . . 3
                Disk Equivalences  . . . . . . . . . . . . . . . . . 7
                MicroVAX 2000 Comm Ports . . . . . . . . . . . . .  16
                Old Q-bus Hardware . . . . . . . . . . . . . . . .  19
                Printronix 600 Setup . . . . . . . . . . . . . . .  28
                11/70 Tips . . . . . . . . . . . . . . . . . . . .  31
                RK07 Tips  . . . . . . . . . . . . . . . . . . . .  37
                Reviving a PDP-11/24 . . . . . . . . . . . . . . .  39
                Backing Up With RSX  . . . . . . . . . . . . . . .  50
                SCSI for RT-11 . . . . . . . . . . . . . . . . . .  55
                Hard-to-find Parts . . . . . . . . . . . . . . . .  58
                MicroVAX II Battery  . . . . . . . . . . . . . . .  61
                RL02 Spare Lightbulbs  . . . . . . . . . . . . . .  64
                Old Hardware Potpourri . . . . . . . . . . . . . .  66
                About the DECUServe Journal  . . . . . . . . . . .  70
                Contact Information  . . . . . . . . . . . . . . .  70
 

bqt

Veteran Member
Joined
Sep 15, 2019
Messages
506
Location
Zurich, CH
The last remaining 11/70s were performing mission-critical services. Some of the last ones I knew of were at Hydro Quebec (including an 11/74!), various Bell Telephone installations (replaced by WeCo 3B20 systems and later by Sun SPARC systems) and the Los Angeles E911 dispatch system.
The 11/74 at Hydro is a story I've seen many times, and I've tried to chase it down. So far, it seems that one is a rumor. But yeah, lots of the last 11/70s were running mission critical systems. I wonder if any are still in operation...
It is more of an aesthetic thing. There's nothing like powering on an 11/70 with a bunch of factory-fresh fans. It isn't a job for thr faint of heart, though - there are a LOT of fans (32), some of them in very inconvenient places, IIRC.
Yep. And I do agree. It's nice to have everything working and in good order. Always feels better.
You might find DECUServe HARDWARE_HELP 238.x helpful. It is abstracted in http://www.people.vcu.edu/~agnew/MVAX/9412_DECUSERVE_JNL.HTML starting on Page 31. One of the things in there is the last known revision list for all 11/70 components. As I noted there, "This information is especially important as the DEC FCO documentation (DEC-O-LOG, MDS) is *incorrect* and *incomplete* for the 11/70." so other than verifying revisions, I don't know if the sites with archived DEC-O-LOG fiche have the correct info to actually perform the ECO/FCO.
Thanks. But just lists of what revisions boards should be at is good, but not what I was looking for. I'd like to know what needs to be "fixed" to get to a certain ECO/FCO. You can't really order any of this stuff from DEC anymore, so you need to fix it yourself, if needed. Also, it's not always that easy to tell which ECO a board is at. So ways to tell would also be welcome.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
The 11/74 at Hydro is a story I've seen many times, and I've tried to chase it down. So far, it seems that one is a rumor. But yeah, lots of the last 11/70s were running mission critical systems. I wonder if any are still in operation...
I have no idea if the 11/74 system is true. But at a DECUS open-mike session someone stood up, announced their name and organization as usual, then asked a question. The DEC answer started "You should really upgrade to an X, you'll save enough electricity in the first year to more than pay for X" and the customer broke in and said "Perhaps you didn't hear my company name - Hydro Quebec". Took a few minutes to sink in.

Thanks. But just lists of what revisions boards should be at is good, but not what I was looking for. I'd like to know what needs to be "fixed" to get to a certain ECO/FCO. You can't really order any of this stuff from DEC anymore, so you need to fix it yourself, if needed. Also, it's not always that easy to tell which ECO a board is at. So ways to tell would also be welcome.
If there's an on-line DEC-O-LOG site, that should have the FCOs. Andover and the other US rework centers were pretty good at keeping boards labeled proprly, with either a letter punch on the back side of the bracket or wrapping a cloth letter label around one of the spacer tabs on metal-handle boards. The only place it gets tricky is where multiple etch revisions had defferent rev levels. Later on, DEC got better and either changed the M number or used a suffix like -YA, etc. to designate different etch revisions if they were at different rev levels. Most of the time, an etch revision was just to eliminate making more boards and having to manually ECO them, so a board at, say, rev E might either be an older etch with a bunch of ECOs or a newer etch with those ECOs incorporated. It's usually when you have a design that is un-salvageable (RQDX1, anyone?) where things get wonky.

I believe Alan Frisbie ended up with my fiche collection. He retired and moved some time ago, but I think he's still reachable at the usual email address.
 

bqt

Veteran Member
Joined
Sep 15, 2019
Messages
506
Location
Zurich, CH
I have no idea if the 11/74 system is true. But at a DECUS open-mike session someone stood up, announced their name and organization as usual, then asked a question. The DEC answer started "You should really upgrade to an X, you'll save enough electricity in the first year to more than pay for X" and the customer broke in and said "Perhaps you didn't hear my company name - Hydro Quebec". Took a few minutes to sink in.
Yeah, that's the story I've heard too.
However, speaking to DEC people who worked on the 11/74, they claim that all were returned after the field trial ended. No machine were left out at any customers. And I have not managed to find any physical evidence of any 11/74 at Hydro Quebec by asking around. But that story have popped up several times. You were active in DECUS, Terry. Were you perhaps at the session where this exchange took place? It would be very interesting if I can even find a first hand witness who heard it...

And I've spoken a lot with Dave Carroll, who kept the last DEC internal 11/74 running until about 2001.
If there's an on-line DEC-O-LOG site, that should have the FCOs. Andover and the other US rework centers were pretty good at keeping boards labeled proprly, with either a letter punch on the back side of the bracket or wrapping a cloth letter label around one of the spacer tabs on metal-handle boards. The only place it gets tricky is where multiple etch revisions had defferent rev levels. Later on, DEC got better and either changed the M number or used a suffix like -YA, etc. to designate different etch revisions if they were at different rev levels. Most of the time, an etch revision was just to eliminate making more boards and having to manually ECO them, so a board at, say, rev E might either be an older etch with a bunch of ECOs or a newer etch with those ECOs incorporated. It's usually when you have a design that is un-salvageable (RQDX1, anyone?) where things get wonky.

I believe Alan Frisbie ended up with my fiche collection. He retired and moved some time ago, but I think he's still reachable at the usual email address.

Unfortunately, I don't trust whatever markings I have on cards or backplanes. Not only can these things fall off with age, I'm also not sure how well those markings were even kept up to date on boards around here. Which is why I am very interested in the itty gritty details of all ECOs.

I might try pinging Alan then... Thanks.
 

DDS

Veteran Member
Joined
Sep 5, 2013
Messages
589
The last remaining 11/70s were performing mission-critical services. Some of the last ones I knew of were at Hydro Quebec (including an 11/74!), various Bell Telephone installations (replaced by WeCo 3B20 systems and later by Sun SPARC systems) and the Los Angeles E911 dispatch system.

In the early to middle 1980's I worked in and around a couple of those "mission-critical" systems, BellSouth's North Dade SCC (Switching Control Center), the South East Regionial Data Center's Minicomputer Maintenance and Operations Center (SERDAC-MMOC), and then at the Miami Network Operations Center (NOC) after BellSouth reconstituted the SCC's into NOC's. While at the MMOC I worked with the System Administrator for the SCCS network that supported the 5 SCC's in South and Southeast Florida. The SCCS allowed technicians at the SCC's to remotely control and to a certain extent maintain the (at the time) #1ESS, #1AESS, #5ESS, and DMS100 switches that provided local and tandem telephone services in the area. It has been a while and some of the details have faded, but I do remember there were 33 of those switches in the Miami-Dade County area alone.

At that time, BellSouth had already transitioned from the #2SCCS system using PDP11/70's in the MMOC, PDP11/23's in the SCC's, and a whole lot of discrete data circuits to tie the whole works together. That had been replaced with the #3SCCS system reusing the PDP11/70's but replacing most of the rest of the network with 3B5 support processors, Datakit2000 and Datakit500 units in the MMOC and SCC's, and 56kb trunks to tie the new network together in a StarLAN configuration. New growth as the remaining electro-mechanical switches (#5XBAR) were converted to more modern hardware was supported by 3B5 host processors in the MMOC. Plans were in the works to replace the PDP11/70's with 3B5's, but by the time I transferred to the MMOC, none of those conversions had been started.

Shortly thereafter, a decision was made that the 3B5, which was intended to support 150% of the capacity of a PDP11/70, was not really up to the task, and the two 3B5's in our shop were upgraded to 3B15's. After I did a 5 tape "EPOCH backup", a processor tech would replace the processor, the memory, and then I assisted in replacing the two CDC built hard drives with larger 340MB units. Then the processor tech would run the system diagnostics, I'd reload the "new" 3B15 from the backup tapes, and voila, the trick she was done. Later I convinced our Senior Engineer that most of the complaints we were receiving from the SCC's were due to under capacity, and we received 3 brand new host processors that went in as 3B15's. Only after that did the process of cutting PDP11/70's over to 3B15's begin. I left for the Miami NOC around 1986 and can't speak for what happened in the rest of what had been "The Bell System", or what happened at SERDAC after I transferred out, but the PDP11/70's that I saw replaced were replaced with 3B15's. We used a ton of 3B20D's (the high reliability duplex version) as Attached Processors in the #1AESS switches, and as Administrative Modules in the #5ESS switches, but I never saw a 3B20 at the MMOC during my tenure there.
 

DDS

Veteran Member
Joined
Sep 5, 2013
Messages
589
I am sometimes amazed/confused by places that have an 11/70, and seem to have such problems keeping it running. Admittedly, we do have lots of spares at Update (we actually have three 11/70 machines), but if needed, I'm pretty confident I could get them all running perfectly fine.

That brought a chuckle.

I, as an Electronic Tech, was not allowed to even open the doors of the 11/70's or 3B15's I did the software on. Union rules and all that. When one of them was misbehaving, it was up to a Processor Tech to work on it. That rarely happened with "my" machines, but i did get to know the processor techs quite well over the years, and knew that when the rubber really met the road there were only 2 of the 15 or so who could fix just about any 11/70 trouble. "My" 10 machines ran very well, but just across an aisle was the COSMOS section, and they had a real "problem child." Remember the poem?

There was a little girl,
With a pretty little curl,
Right in the middle of her for'ed,
When she was good,
She was really really good,
But when she was bad she was horrid!

One afternoon as I was leaving I saw a processor tech setting up to work on "problem child". I gave him a wink and said "good luck". He replied "I know, right?" Next morning, as I walked in, there he was, sitting on the floor, at least 16 hours with no sleep, bags under his bloodshot coffee overdosed eyes, most of the guts of that 11/70 spread out around him on the floor, with an irate COSMOS System Administrator screaming at him, "I need that machine up. I want it back right now!"

So he got up, picked up his lunch box, said "OK it's yours" and walked out the door.

So there she was, all of 5 feet of fury and frustration standing amid the parts of "her" machine.

That was the talk of the shop for quite a while.
 
Last edited:

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
You were active in DECUS, Terry. Were you perhaps at the session where this exchange took place? It would be very interesting if I can even find a first hand witness who heard it...
The only thing I was present for was the tech Q&A where someone asked some question (don't remember what, but my ears would have perked up if it had "11/74" in it, so probably unrelated) got DEC to tell a power company how to save on electricity costs.

Unfortunately, I don't trust whatever markings I have on cards or backplanes. Not only can these things fall off with age, I'm also not sure how well those markings were even kept up to date on boards around here. Which is why I am very interested in the itty gritty details of all ECOs.
DEC switched to board-level swaps when Data General was still replacing ICs on boards at customer sites. Unless the FCO was something simple like "replace ROMs" or "move jumper", the board generally got swapped with an up-to-rev one and returned to the depot for rework. Here in the US (at least at my site) even boards that were supposed to be FCO'd by replacing a zillion ROMs got replaced at the board level to save time. I think the CI780 had a whole forest of them, and the firmware people went out of their way to avoid changes that would shift addresses around, so the minimum number of EPROMs needed to be changed, even at the expense of spaghetti code. Field Service used to dread new revisions of the VAXcluster Compatibility Matrix as it meant that they would need to visit a site with a crate full of EPROMs to change parts in the CI780, HSC50 (and port cards) and install new CI microcode on the console floppy (slow) or the TU58 (glacial), so they tended to come out with boards and just wholesale swap them, give the customer the new microcode on some media, tell the customer to install it, and run away ASAP.

Anyway (post-digression), if your modules have a revision stamped on the handle or wrapped with a cloth letter label, it is pretty much guaranteed that the boards are at least that revision (but could be higher). The reason the FCO documents are incomplete (as noted in my 32-year-old DECUServe post) is because the changes weren't supposed to be performed in the field. So the FCO just says "Swap board Mxxxx for revision x or higher", or sometimes doesn't mention a particular issue but there's an unrelated document that lists revision levels, with no indication of what or why.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
In the early to middle 1980's I worked in and around a couple of those "mission-critical" systems, BellSouth's North Dade SCC (Switching Control Center), the South East Regionial Data Center's Minicomputer Maintenance and Operations Center (SERDAC-MMOC), and then at the Miami Network Operations Center (NOC) after BellSouth reconstituted the SCC's into NOC's. While at the MMOC I worked with the System Administrator for the SCCS network that supported the 5 SCC's in South and Southeast Florida. The SCCS allowed technicians at the SCC's to remotely control and to a certain extent maintain the (at the time) #1ESS, #1AESS, #5ESS, and DMS100 switches that provided local and tandem telephone services in the area. It has been a while and some of the details have faded, but I do remember there were 33 of those switches in the Miami-Dade County area alone.
That brings back memories. I believe we had the largest 1AESS ever deployed (201-332), with 192000 WTN's. It was the second test switch for Caller ID (before it had been named) and for some reason they decided to retrofit every WTN with the modem. [For those playing along at home, the 1A was a computer-controlled relay-based switch, so adding things like this involved actual hardware. The #5 ESS was a digital fabric switch, so this sort of thing was a matter of firmware. WTN is Working Telephone Number, essentially a copper pair per number except for certain trunked numbers, as opposed to BTN which is the Billing Telephone Number.]

They had a #5 ESS waiting to be deployed and it was assigned 201-200 (and therefore hosted the lowest dialable number in the NANP - 201-200-0000). The initial customer was a college down the road from me, and after around 10 PM there was essentially no phone traffic. So the poor #5 would do a self-check and go "Hmmm, I haven't processed a single call in half an hour. I must be broken. HEEELLLLLLPPPPPPP!!!!" and start the maintenance fire drill. And of course it was in an unattended CO. Side note: Have you ever watched a 1A console go "P<bell>P<bell>", etc for minutes on end, hoping it would self-heal and get out of phasing?

Anyway, I told them that there was no way I would accept our prefix being changed over to the #5 until it could mimic the 1A behavior completely (including the ker-chunk line open during call waiting, and a bunch of obscure other [mis]features). That's the reason the "Extended 1AESS compatibility" feature is in the 5E generic to this day. They actually didn't tell me about it when they switched my prefix over and I didn't notice it until I tried calling the voice maintenance port [For those at home, it was a jealously-guarded secret phone number that let you directly interact with the duplex processor controlling the switch - you would enter commands via touch-tone and the switch would use speech synthesis (an extremely low bit rate implementation of Doug McIlroy's TTS code, to reduce the CPU load) and you could do things like open any pair connected to the switch for up to 30 minutes and all sorts of other neat maintenance things.] and noticed it worked for my prefix.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
One afternoon as I was leaving I saw a processor tech setting up to work on "problem child". I gave him a wink and said "good luck". He replied "I know, right?" Next morning, as I walked in, there he was, sitting on the floor, at least 16 hours with no sleep, bags under his bloodshot coffee overdosed eyes, most of the guts of that 11/70 spread out around him on the floor, with an irate COSMOS System Administrator screaming at him, "I need that machine up. I want it back right now!"

So he got up, picked up his lunch box, said "OK it's yours" and walked out the door.

So there she was, all of 5 feet of fury and frustration standing amid the parts of "her" machine.

That was the talk of the shop for quite a while.
I've got a goodie for you. The DACS [for those at home, digital cross-connect and multiplexing for T1s into T3s] serving my facility (which had a few hundred T1s, a dozen or so T3s, an OC-12 and an OC-48) had been having problems where it couldn't be backed up. As you may know, it used an annoying WeCo implementation of QIC tape format (same mechanism as the Dimension PBX). After many months of failing to back up, they got a Lucent (by then it was Lucent) "tech" out to look at it. The DACS says something like "Failure writing tape - do you wish to initialize (y/N)?" and the hapless tech typed Y, not realizing it meant initializing the DACS, not the tape. The entire configuration was erased, leaving it a very expensive paperweight and doing all sort of things like taking most of E911 offline, not to mention interoffice trunks. I didn't find out about that until later, though. All I knew is that all of my T1s went down along with some of the T3s.

I pick up my office phone to call one of the people I knew who worked in HiCap. It took several seconds to get a dial tone (very unusual for a 5E) and I got a reorder tone immediately after dialing the prefix. So I grabbed my cell phone and went outside (my datacenter was a very effective Faraday cage) and called the HiCap guy's personal cell number. He answers and I ask him if there is anything unusual going on, and he says no. I ask him to pick up his office phone and call anyone, I'll wait. He comes back and says something like "Holy $hit! I'll get back to you!" and hangs up.

After several hours most of our T1s and all T3s were back up, but I still had a few dozen T1s out. Oddly, they were coming back in what appeared to correlate to their FOC date (installation/provisioning date). After a few days, they were all back. I eventually asked my HiCap guy what the story was, and he told me the above bit about the tape drive, and that they had to restore the DACS configuration from a half-year-old backup tape and then manually input each subsequent service or disconnect request from their records. That's why it took so long to get everything back - people were frantically typing on consoles.

I have another story about having an interfloor T3 problem in Manhattan's Broad Street CO, and went downstairs to the floor housing the transmission equipment. I asked the tech on duty to enter a change request to change the LBO [Line Build-Out / signal level] on that T3. I expected her to make a note of it. Instead, she looks up some info and says "come with me" and we went into another room filled with transmission equipment. She goes to a FLM2400 and counts cards, and proceeds to pull one out, flip the LBO switches, and put it back in. Of course, my T3 was only 1 of 4 served from that card. I wonder what the other 3 customers thought when their circuits dropped out for a few minutes.
 

DDS

Veteran Member
Joined
Sep 5, 2013
Messages
589
Side note: Have you ever watched a 1A console go "P<bell>P<bell>", etc for minutes on end, hoping it would self-heal and get out of phasing?

"I knew you were going to bring up something unpleasant" -- Edith Anne (Lilly Tomlin) in a "Tales From The Big Chair" skit on "Laugh In"

In the 60's when the #1ESS was being developed everybody was trying to figure out how to store information magnetically. Magnetic tape and core memory were only two of the solutions. Western Electric had their own ideas. Enter the #1ESS Program Store. Think core memory that is read only. The magnetic part was actually small (0.030 inch) squares of soft iron material glued to a fairly large aluminum card maybe 6 x 12 inches and quite thin. Transient data and changed data were written to an 8K word Call Store that would over time fill up. Then a tech, actually several techs, would have to do a multi shift process known as an office update and merge the data from Call Store into Program Store.

You can see parts of the process beginning about 16 minutes in the below video.

https://www.youtube.com/watch?v=1G2keZBSZmY

So after 6 months of schooling and another 6 months of OJT they decided I could be trusted on my own. My boss handed me a load package that consisted of starting the update in the Indian Creek office on Miami Beach (MIAMFLIC86E). So I did. I won't further bore you with the setup details but about half an hour in I knew things weren't going to plan. It took me about 20 minutes to write the first "mod" of cards, which was normal, but they refused to "verify" which was not. An hour in I got out the practice, opened the special update tool kit, and more or less figured out the card writer was out of alignment. So I called my boss who tore me a new one because the card writer had just been aligned by his favorite "fair haired wonder tech" and couldn't possibly be the problem. He was not happy when he showed up about 30 minutes later and started double checking my work. One of the tricks to get a mod to verify was to make sure the cards were fully seated in their slots. You could gently tap them with the meaty part of the side of your hand but if you hit them too hard it was possible to drive one of them through the read out tape that multipled through the back of the Program Store module itself. He was even less happy after he did exactly that. The Model 35 TTY did exactly what you described as it printed out a Phase 1, rolled into a Phase 3 and we heard that big KERCHUNK when every cutoff switch in the office opened up and we had 35,000 customers more or less out of service. But we weren't out of the soup yet. The switch rolled from the Phase 3, into a Phase 4, and then a Phase 5, and stuck there rolling Phase 5's. I looked at the boss and he was actually sweating. I told him we were probably going to have to manually initiate a Phase 6 (things can only get one step worse). He got really red in the face and left.

It was already broke, my boss had left me twisting in the wind, i figured it was likely my last day with Ma Bell, so what the heck. I reviewed my notes, configured the MCC to request a Phase 6 and pushed the execute button. Things got really quiet for a few seconds but then it came up mostly happy. i quickly put the office back into simplex since I knew the "supposed to be duplicate" halves of memory couldn't possibly match, called ESAC (first tier tech support) in Atlanta, told them what had happened, did what they said to do, and beat feet back to the center.

As I lugged the 60 pound update tool kit past the break room I saw the boss and the wonder tech with their heads together whispering excitedly until I entered the room. Figuring what was likely happening I calmly informed both that the only thing they could possibly do to make our situation worse was if they tried to blame the entire Charlie Foxtrot on me, the rookie.

Western Electric had to come out on an emergency basis and replace that entire Program Store shelf. Wonder tech decided that the Indian Creek cad writer was obviously defective, though it had worked like a champ before he volunteered to align it. So he went to another Central Office and "borrowed" the card writer out of that switch. Long story short, the center now had two switches with no working card writer and Call Stores that were either full of damn close to it. Oh, and one wonder tech with a slightly tarnished halo.
 

Agent Orange

Veteran Member
Joined
Sep 24, 2008
Messages
6,169
Location
SE MI
That brought a chuckle.

I, as an Electronic Tech, was not allowed to even open the doors of the 11/70's or 3B15's I did the software on. Union rules and all that. When one of them was misbehaving, it was up to a Processor Tech to work on it. That rarely happened with "my" machines, but i did get to know the processor techs quite well over the years, and knew that when the rubber really met the road there were only 2 of the 15 or so who could fix just about any 11/70 trouble. "My" 10 machines ran very well, but just across an aisle was the COSMOS section, and they had a real "problem child." Remember the poem?

There was a little girl,
With a pretty little curl,
Right in the middle of her for'ed,
When she was good,
She was really really good,
But when she was bad she was horrid!

One afternoon as I was leaving I saw a processor tech setting up to work on "problem child". I gave him a wink and said "good luck". He replied "I know, right?" Next morning, as I walked in, there he was, sitting on the floor, at least 16 hours with no sleep, bags under his bloodshot coffee overdosed eyes, most of the guts of that 11/70 spread out around him on the floor, with an irate COSMOS System Administrator screaming at him, "I need that machine up. I want it back right now!"

So he got up, picked up his lunch box, said "OK it's yours" and walked out the door.

So there she was, all of 5 feet of fury and frustration standing amid the parts of "her" machine.

That was the talk of the shop for quite a while.

Well, don't keep us hanging. Did you get it fixed and what was the problem? :whaasup:
 

DDS

Veteran Member
Joined
Sep 5, 2013
Messages
589
Well, don't keep us hanging. Did you get it fixed and what was the problem? :whaasup:

Not really clear on which "it" you're asking about.

1. That problem 11/70 was a machine belonging to another department. I was SCCS. The sick machine was COSMOS. Plus hardware was someone else's ricebowl. But as far as I know that 11/70 was a problem until I left for another building a few years later.

2. Somebody eventually fixed both card writers, but it wasn't me. One of the weak points of the SCCS environment, as it was implemented by the old AT&T, was that no ESS tech was assigned to an office. The techs who were assigned to the center rarely left it. Some of them analyzed reports, made trouble tickets, and handed them off to loading and dispatch clerks. Those clerks bundled them up into load packets, and assigned them to rover techs who went out to an office to work on a trouble for a given amount of time, wrote the results up, then went on to the next ticket. We rarely got to work on the same trouble until it was cleared. We also rarely went to the same building two days in a row. The incomplete tickets went back to those analyzers who would put suggestions on the ticket as to their best guess on what to do next. Then the ticket would go back to loading and dispatch. It was completely random if you got that same ticket the next day. Note that I didn't mention anyone in the chain that was in charge of getting a trouble fixed. As far as I could tell, no such responsibility existed and it appeared to me that AT&T, the architect of the system, liked it that way. I'm not sure how Southern Bell and later BellSouth felt about it, but as a wholly owned subsidiary, it wouldn't have made much difference. Questioning the system just got you marked as someone with a bad attitude.
 

Terry Kennedy

Experienced Member
Joined
Jul 7, 2017
Messages
241
Location
New York
Those clerks bundled them up into load packets, and assigned them to rover techs who went out to an office to work on a trouble for a given amount of time, wrote the results up, then went on to the next ticket. We rarely got to work on the same trouble until it was cleared. We also rarely went to the same building two days in a row. The incomplete tickets went back to those analyzers who would put suggestions on the ticket as to their best guess on what to do next. Then the ticket would go back to loading and dispatch. It was completely random if you got that same ticket the next day. Note that I didn't mention anyone in the chain that was in charge of getting a trouble fixed. As far as I could tell, no such responsibility existed and it appeared to me that AT&T, the architect of the system, liked it that way. I'm not sure how Southern Bell and later BellSouth felt about it, but as a wholly owned subsidiary, it wouldn't have made much difference. Questioning the system just got you marked as someone with a bad attitude.
This wasn't limited to CO work - the same idea was mis-implemented for customer rolls. Each tech ("craft") would get a big printout (Model 40 with a bad ribbon, printed on recycled toilet paper) with their jobs for the day. Crafts got penalized for any incomplete tickets at the end of the day. So at the start of shift, you'd see everybody sitting in their trucks tearing the jobs into individual sheets and sorting the ones they thought were hard to the bottom of the pile. Why get to your 2nd job of the day and get stuck there for hours and hours, and turn in a big pile of jobs as incomplete when you could do all the easy ones first, then just hand back the dog jobs at the end of the day? The end result was that anything that was complicated simply never got done, and you could have something in the dispatch queue for weeks if you were really unlucky. Then one day the Great Equalizer was introduced - to save money, they eliminated the dedicated numbers for things like Assignments. For example, you'd get a work order that said something like "provision service from pair 23 on the 806B cable from pedestal 11-25". You'd go up the pole with your test set and discover that the pair you were given to use was apparently open ("apparently" because in a previous cost-saving measure, you no longer had a kick meter to evaluate pairs). Before the Great Equalizer, you'd simply call up the Assignments desk and say "I have an open on the assigned pair for order XXYYZZ123" and they'd tell you what pair to use, update the docs, and you were good to go. But now you no longer had direct access and had to dial the same 611 repair number as customers, and sit on hold for half an hour or more listening to "Your call is very important to us..." over and over. So there was no longer any such thing as a guaranteed easy job.

Another brilliant idea, when fiber began to be widely deployed and not just between COs, was that the splice techs would go up in the cherrypicker with the fusion splicer. Rumor has it they dropped over $500K of fusion splicers in the first week alone. So the Great Plan was changed to leave large coils of fiber (splice loops) on the poles and switch the splice techs to non-bucket trucks. When they arrived at a job, they'd call Dispatch for someone with a bucket truck to come out and drop the loops so they could splice from the comfort of their trucks. When they were done, they were taught to tape (or use those 1/2" fiberglass mat cable ties) the whole mess to the pole(s) at eye level. Then, "supposedly", anyone with a bucket truck who happened to be driving past and spotted this was supposed to stop and re-hang the fiber and splice box back at the top of the pole". See no evil and all that - stuff never made it back up in the air unless someone called and complained.

The one nice thing about the CO rover system was that the same badge worked for both the office(s) you were normally in as well as (at least in my case) every other CO in the LATA. At first we had these really clunky blue Westinghouse Security Electronics badges which were actually a thick PCB sandwiched in between protective plastic layers. If you sat down on it, it would crack. Then they went to more standard thin RFID badges, still separate from the photo ID. Here's me with both older and newer access card types (DNY is "Downstate NY" or LATA 132):

TMK%20Verizon%20fronts.jpg

TMK%20Verizon%20backs.jpg


Low resolution and defaced to discourage copying. Also a 15+ year old badge design missing the current holographic text. And I had more hair. :p

The other great thing about these badges was that before 9/11, if you needed a restroom break, you could go just about anywhere and go "Telephone repair" and they'd let you in without question.
 
Top