The crowd gathered around for an impromptu ceremony. This was a rare occcasion where the evening shift people who worked 4PM-2:30 AM were asked to come in for 3 PM. The dayshift finished at 3 and the 1 hour transition was normally a cleanup period. This was a special day as we shared in the announcement of one of the 25-year employees retire from the factory. She had worked the same riveter machine for most of the last 5 years, which was for securing aluminum window frames for pop-out windows in Ford and Chevrolet vans.
It was amazing to watch the joy and emotion as she was presented with a completed, glazed window, sealing inside it a printed picture of her at her riveter. Proudly holding up a picture of her grandchildren, she smiled and held back tears as she knew she would spend lots of time with them starting the following Monday of her retirement. The career started as a job. A temporary gig she took in her mid to late twenties to hold her over and save up for potential college. A temporary gig just like the one I took 18 months earlier here at the same factory at age 19.
At the end of the celebration, I made my way to the door. My time card was punched in at 3:25 and out at 3:50, never to be punched again. I would get on my bicycle for the 40 minute ride home and call Human Resources from there. With enough money in the bank for just a bit more than the next month’s rent, I quit. I vowed to the HR person that I would never retire from a factory and thanked them for their help and support over my 18 months of work.
It wasn’t about some new dream. It wasn’t about not being happy for those who would retire from a factory job if that made people happy. I just knew it wouldn’t make me happy. I didn’t even have another job lined up. All I had was a countdown to the end of my savings and a desire to do something more.
Make every day a day you decide whether this is the thing in your life you would be happy looking. back on.
23 Things Only 90s Sysadmins will Remember
Sometimes it’s a beautiful combination of laugh-inducing and cringeworthy as we look back on the start of our careers in IT. University and College students starting this September will have grown up entirely post-Y2K. Having grown up in enterprise IT through the 90s, I bet that you can share these things that will make you smile a little.
Feel free to drop in comments with your own fun memories of technology in your career!
That’s right, folks. Ethernet wasn’t the first networking that was adopted in the enterprise. As a 90s administrator, you’ll remember the pre-switching days and the ever-enjoyable experience of chasing down beaconing devices on a network as the token made its way downstream. The physical click as a network card lit up the port on an MAU (or MSAU for the traditionalists of the Multi-Station Access Unit). Those nifty little tokens flew around the physical and virtual ring at 4 or 8 Mbps which felt like a jet car compared to the BNC serial networking before it. The late 90s brought switching and 16 Mbps with a protocol battleground laid out, to be won over by the Ethernet adoption at 10/100 Mbps and a much lower cost.
Enterprise computing needed enterprise computers. The price tag on an IBM PS/2 (aka Personal System 2) was steep, but the easy to replace parts and enterprise support agreements with many IBM-centric shops (aka Big Blue customers) were swimming in stacks of the IBM PS/2 systems. You also remember the Micro Channel architecture and that popping feeling as you pushed in the floppy drive ejector. Classic days!
Autoexec.bat and Config.sys files
Gamers will know this challenge especially well. Should we load into high memory? What drivers can I put before the OS and game? What can I strip out of the boot system to let things run faster? This was also the first appearance of MS-DOS getting a login screen as each nerdy sysadmin mastered the science of ASCII art to welcome their users to the freshly booted system.
Novell certifications were hugely valued
VMware wasn’t even on the radar (only to be launched in 1998), and Microsoft was still finding its feet in the certification space with NT4 only making a slow transition to Windows 2000. Novell dominated the file server and identify management (+1 for NDS, always) environment of many entrerprises. Even the small business servers were often running Novell NetWare and the badge of honour was the length of uptime you could have on a server. Administrators flocked to the CNA (Certified Novell Administrator) exam and then upped their game with the challenging and more rare CNE (Certified Network Engineer).
Brain dumps for certifications
Certifications were becoming a hot commodity, and so were brain dumps. These were online repositories of “hypothetical” questions (often word-for-word renditions) and multiple choice answers with the right one chosen for you. This led to the addition of some strongly-worded NDA statements as you sat for a technology certification, and also the familiar 8.5×11 whiteboard and box of kleenexes in every exam cubicle. Some say the kleenexes were for the tears of sysadmins who didn’t make the cut, rather than to wipe the dry-erase board.
As Microsoft Certified Professionals poured out of certification centres at an incredible rate, we also saw a large number of tech pros leaning forward to the full MCSE (Microsoft Certified Systems Engineer). The issue among the tech community was that certification schools and boot camps were popping up like wildfire and producing dozens to hundreds of MCSE recipients in a matter of days using tools like brain dumps and teaching 100% towards test materials. This led to the industry (mostly old CNE techies) labeling these as “paper” MCSEs which meant they only got the cert but lacked any real experience.
Deploying Windows 95
There were actual videos featuring the cast of Friends at the launch of Windows 95. This also led to pain an suffering among the joy of successful deployments company wide for sysadmins everywhere. The standardization of the corporate desktop became a new possibility thanks to ADM files, imaging processes and more. This was the dawn of a GUI-driven enterprise with no turning back.
This was actually an acronym as GHOST (General Hardware-Oriented Systems Transfer) which started as a PowerQuest tool, later bought by Norton, and eventually Symantec. Using a nifty tool to clone hard disks meant speeding the deployment of DOS systems and eventually Windows systems (thanks to the magic of SID regeneration in later versions). Sysadmins finally learned the value of automation and got away from the disk install or network installs of operation systems.
This is for the print shop admins who spent time in those Big Blue customers. You may have even administered a BARR print system at one time and learned of the oddities and challenges of converting print protocols. EBCDIC (Extended Binary Coded Decimal Interchange Code) was the peripheral code standard and eventually gave way to ASCII as distributed printing became cheaper and more popular. It also cut down on the number of print paper carts rolling through the offices all hours of a day and probably ended a few mail room jobs.
Replacing Coax with Cat5
BNC, yeah you know me! Coax and BNC was the early peer-to-peer networking wire of choice (or lack of choice) and was quickly giving way to the use of Cat5. We quickly bypassed Cat3 for the Cat5 UTP (Unshielded Twisted Pair). I bet you can still recite the wire colours and the order to create a crossover cable with your own punchdown kit.
Life comes at you fast in networking. At the same time, there was nothing more popular in the help desk and desktop support office as a lap link cable. For those who enjoyed the slow, reliable, parallel-to-parallel port goodness to transfer data between machines, Laplink software and associated crossover cables were your friend. By the 90s, these became rare, but I’m betting you held onto your cables a little longer than we all should have just because of nostalgia.
Lotus Notes and Novell GroupWise
Not only was I a Certified Lotus Notes Administrator, and Certified Lotus Notes Developer, but I was one of a handful of GroupWise-Certified Novell admins around town.
We were in the heart of a war over the de facto email standard, and productivity software was the new panacea. Lotus Notes possessed integrated email, directory services, Internet email, and applications (known as databases…much to the chagrin of legit DBAs). If it wasn’t for wide proliferation of HTML applications and rapidly adopted internet standards, you’d probably still be logging into a Lotus Notes client today.
Java takes over the development world
Write Once; Run Anywhere. That was the tagline and mantra of the Java developer. Write once; run slowly, and probably on the wrong version, anywhere, was the tagline and mantra of sysadmins. Java was fighting the Microsoft suites for dominance in the enterprise application market. Little did we know that the often despised (by sysadmins at least) language which was rapidly evolving, would continue it’s growth AND still leave a wide door open to .NET and the Microsoft tools.
The only pizza boxes in a 90s data centre were the empty ones left by sysadmins after an overnight upgrade of Windows NT or a Novell Server. While the Unix folks logged into their Sun Microsystems servers remotely, the rest of the Wintel kids were piling up server towers by the dozens along two-tiered shelves in every server room. Rack servers became more prolific in the later 90s, and we bid good riddance to the mini-towers of every shape and size.
Installing Windows 3.1 from multiple 3.5” diskettes
Many operating systems were so large that they spanned multiple diskettes. Those pre-CD days were not fun. I’m guessing that you’ve held onto a stack of 3.5″ diskettes, carefully sequenced for an installation. This also led to the fun of having 40 of 41 diskettes and not being able to install. Disk 31 was like the proverbial sock eaten by a clothes dryer for a sysadmin.
Hey, it was pretty cool to her able to cram 100 MB on a single piece of physical media. As you inserted the almost Nintendo cartridge shaped cassettes into the Zip drive and it clicked and whirred, you smiled your nerdiest smile knowing you could fit 70 floppy diskettes on a single Zip drive. You felt like a storage king…until the CD-R and CD-RW became easily affordable, and much faster…which didn’t take long.
Laptop peripherals were as awful as you could imagine. Being expensive, difficult to carry, or just unavailable was the biggest challenge of the enterprise laptop user. How could you have a modem, an ethernet card, and a sound card in that Toshiba laptop?! Easy! Carry a stack of PCMCIA (or PC Cards) in your laptop bag to swap out as needed. Ahhh the smell of early standards.
The magical sound that indicated a 56k connection
Forget the pain of not having Google Fibre. How about crossing your fingers that you get the full rocket-like speed of 56k versus 33.6 which was triggered by that telltale sound in the midst of a modem handshake. The next generation of Internet users will only know the pain of 3G versus LTE and believe that Edge is abysmal (which is nearly 4 times the bandwidth of a 56k connection).
High speed networks in the data center began with T1, Fractional T1, and if we were lucky, upgraded to a T3. With a T1 being around 1500$ per month, it’s hard to imagine that a 1.544 Mbps line can even compare against today’s 49.99$ per month for 5 Mbps residential internet connections.
The mainframe is the heart of most of our financial, insurance, and retail businesses. As we moved to the next generation of distributed computing and Windows desktop operating systems, we needed to install emulators for the clunky (yet strangely reliable) 3270 and 5250 terminals. Long live the green screen!
The Internet was new
Compuserve and AOL were the leaders in delivering this magical new thing called the Internet. Curating your experience and filling your mail with physical CDs to win over as many clients in the pay-per-hour fight for subscriber dominance. Ironically, by the end of the 90s, AOL would buy Compuserve and most of us were far beyond the initial stages of Internet access. Google would launch in 1998 and change things greatly.
All Software being named 2000
We knew that the decade was closing out. It also caused a sudden and unnecessary urge for software makers to appear leading edge by naming everything 2000. Windows 2000 is probably the most memorable of these. As we look back on products that released late or didn’t last beyond that landmark year, the idea of attaching a release year to the product name is definitely a fairly bad idea. Somehow Microsoft is one of the only ones who holds onto this nomenclature to this day.
Celebrating NYE 12/31/1999 in a data center
Nobody had any idea how the landmark New Year’s Eve would be spent a decade earlier. Little did we know that the lustre and shine of a career in IT would also mean toasting your fellow sysadmin with a bottle of Baby Duck in plastic cups at the office during a short 15 minute break watching the countdown to midnight. This led to worldwide checks of patches and fixes which were the result of a decade of work after someone, somewhere doing what we all do in IT…going to production without gathering requirements first. Besides, who would have thought we would need more than two digit years, right?!
Oracle Ravello Blogger Day 2 #RBD2 – Oracle Cloud Infrastucture
Oracle is driving hard towards customer-focused cloud. The amount of listening being done by the product teams is showing in the product roadmap that we saw at the (Ravello Blogger Day 2) #RBD2 event in Redwood City, CA this week. This event features a broad array of industry folks who are in vendors, ISVs, VARs and other roles.
Oracle was presenting the Oracle Cloud Infrastructure and Oracle Ravello platforms over the course of the 8 hour event. Q&A becomes the dominant and most exciting part of the day as the content is unfolded for us. Some may quip about the “leading from behind” approach, but there can be little doubt that the boldness of Oracle is proving to be more than all talk as the customer examples and successful execution of the OCI story continue to roll in.
Cloud Truly Ready for the Enterprise
SLA is king in the Oracle cloud. Moving mission critical apps to the cloud rather than rearchitecting to map to the cloud patterns is a big driver for Oracle Cloud adoption. While the cloud pundits may shun this approach, the reality is that customers will adopt cloud to get the new features and are comfortably beside the “lift and shift” application stacks. The marketing of this option has resulted in a change from “lift and shift” to the newly dubbed “move and improve”.
Providing versatility and SLA on the OCI service stack is the icing on the typical IaaS cake. Oracle clearly pins their value on the performance side of things, which is top of mind as folks have tested the cloud waters and seen the inconsistencies in performance at many layers of some environments.
I get the feeling that a distinct advantage for Oracle is the fact that their existing customer base will tend towards being enterprise already. Not many Mom and Pop shops are running Oracle RAC. Being able to sell to a strong base will be extremely helpful in early expansion.
Driving SLA and quality of service as the key performance indicators will draw the eye of any good CIO. There is also a surprising humility among the folks that I’ve spoken with here at the event as they acknowledge the need to shake a potential perception problem with leaning into Oracle as a primary provider.
Build, Buy, Integrate
Starting from 12 years behind compared to alternative public cloud providers means being aggressive. The story is playing out on a few fronts. As most of you know, when you can’t build it, you have to buy it. More importantly, you have to integrate it.
Dyn, Ravello, and more are being quickly integrated to widen the portfolio of Oracle Cloud services. Adding Wercker into the portfolio also brought over some development credibility as a nice way to drive the CI/CD adoption on the Oracle Cloud stack.
Filling up the full stack is also particularly important. The evolution and goals are very interesting to watch. The need to present more than core IaaS is apparent. I’m especially excited about the Terraform integrations (you had me at Terraform), and Terraform services (state management, registry and more) that was discussed. I’ll share more on that once I dive further into it.
Further blogs will dive into some of the key areas of Kubernetes and PaaS/CaaS options. I felt those need a bit more of a deep dive.
Did I Mention Performance?
As someone who’s in the business if workload performance through Turbonomic, this resonates deeply with me. The Oracle team is clearly laser focuses on performance and looking to hit those performance targets while being cost-competitive to alternative offerings.
Oracle’s business is full-stack. The goal (clearly stated and fully transparent) is to go directly against the competitors (read: AWS and Azure) with performance and price. Having the customers using their stack already on-premises makes the transition to Oracle Cloud rather attractive. This is the same approach that Azure is taking. Credits and discounts will come into play to make the first steps enticing, but what will keep the customers in-platform will be performance, cost, and service breadth.
It will be very interesting to see the results of diving into the latest iteration of OCI. Look for much more in the coming weeks as I share my experiences.
Blameless Culture: The Path to Agility in IT
You’ve inevitably heard the word blameless used in the context of technology and tech culture. The foundation of DevOps and agile and many other methodologies is around blameless post-mortems. Retrospective meetings and recaps that allow for trust and honesty when discussing what went wrong and what went well.
Here’s the issue: being blameless is really difficult. Like, really, really difficult.
Battling Human Nature
Blameless meetings in many organizations are more like “don’t blame me” meetings which erodes the very foundation of what blameless culture is all about. A societal challenge is getting past the “Not it!!” mentality and embracing the issues without seeking a person to pin it on who isn’t ourselves. People often mistake not taking responsibility with being blameless. Just because you don’t assign blame, constantly un-assigning yourself from it is not a blameless approach.
It’s not easy to admit we made mistakes. Trust me…
A Personal Story: “Wait, that was production?!”
It was 9:30 AM on a Tuesday. Two Active Directory administration windows open, one for production, and one for the test domain with the identical OU structure (to look just like production). As I clicked the option to upgrade the Active Directory version to Windows 2000 native mode and proudly clicked the Change Mode button (clearly stating “this operation cannot be reversed”) and the following acknowledgement that it was irreversible, I sat back in the chair and smiled…for a moment.
Then I checked the other window to what I thought was production…and saw the domain version was mixed mode (legacy). But then I also noticed the domain name was the test domain. I had updated production in the middle of business hours during the potentially busiest login period of the day.
I stood up and took a lap around the cubicle. Once I stepped back in and confirmed it had happened, I called my systems architect who worked with me. “James, I made a mistake. I just updated production instead of test.”
James calmly said “Ok. Let’s get some folks looking at it and figure out if anything happened and what our options are if something broke”. James understood blameless culture. We survived the change (without issue, luckily) and I learned that rallying the team around an issue was better than seeking blame. We would work together on other issues that did have far reaching effect at other times, and the blameless culture and teamwork got us through those events.
Hire the Person that Made a Big Mistake – They Won’t Want to Make Another One
Making mistakes and learning to change process and recovery procedures meant building better IT. As a systems architect and operations team member, our culture grew stronger under blameless acceptance of issues. I also witnessed the very negative results of not using this tactic. A manager at one time told me that even though we had a large, preventable situation occur from human error, we don’t remove the person. We teach them to not have it occur again. If you make the same mistakes again…and again, well, there isn’t a blameless team around who will not take notice and perhaps have to make some changes in your role.
Test-driven development and test-driven infrastructure are wrapped around the foundations of finding the error first and then working back from there. Seek a problem and then find resolution. The same goes for teamwork and production deployments. To move faster, you have to be confident that you are testing and also able to work together as a team without fear of retribution when issues occur. Some issues are avoidable, and some are not. Be blameless before you find the solution and then when you find the cause, accept that it happens…but should happen less. The real leaders then ask “how can we do it better next time?”
Leadership Defined: We Succeed; I Fail
Acknowledging failure or just challenges is healthy, Celebrating success as a team is also extremely positive. My way of encapsulating what it means to be a leader of people in a team goal is “We succeed; I fail”. In other words, celebrate successes as a team, because we all did this together. When looking for where we may not have succeeded, look into yourself for what you feel could have been done better. There may be others involved in the tactical things that went wrong. The best outcome is a learning experience for everyone throughout the team.
Even when things go well, we should always ask how it could have gone better, or what would you do differently? This is the beginning of embracing a culture of experimentation. You have to know that things can fail and we can recover, without blame.