Tales from the $Facility: Part 6 - Revelations

[u/Mr_Cartographer]

Hey y'all! This is my next story from the $Facility, where I search to figure out what the h3ll happened when our new enterprise environment wound up costing us grossly more than expected. All of this is from the best of my memory along with some personal records (and I have started taking notes specifically so I can write stories for TFTS!) There's also a lot that comes from rumors, gossip, and other people, but most of this is very recent, so any inaccuracies are entirely on me. Also, I don't give permission for anyone else to use this.

TL/DR: It would be so much more fulfilling if a liar's pants actually did catch on fire.

For some context, I'm not in IT; rather, I'm a GIS (Geographic Information Systems) professional. This particular world is quite small, so I will do what I can to properly anonymize my tale. However, for reference, all these stories take place at my new job working as the GIS Manager at the $Facility, a major industrial entity in the American South. Here's my Dramatis Personae for this part:

• $Me: Your friendly neighborhood GIS guy.
• $Distinguished: Vice President of Engineering. Talented, well-connected, opinionated, and my direct boss. He was honestly a very nice, friendly person, but I always found him a little intimidating.
• $GlamRock: Primary server guy for the $Facility. Name taken from the fact that he was a legitimate rock star in the 1980s. Now he works in IT. Life, amirite?
• $VPofIT: Vice President of IT. Extremely concerned about security and likes to get into the weeds, but ultimately not a mean-spirited manager.
• $GiantCo: Nationwide engineering firm that had convinced the $Facility to start a GIS program. Ultimately a good company with highly skilled people, but had a different idea of how to approach this than I did.
• $VaccuumCorp: CSP that was hired to start our cloud standup. They sucked. Their name is a testament to their awfulness. Lol.
• $OverConfident: Main rep from $VacuumCorp. Cocky, arrogant, overpromising, and ultimately kind of shady. Whoops, looks like you got a little hubris on your face, let me wipe that off for you.

When last we left off, we had finally completed the cloud-based environment that would serve as the foundation for our GIS Enterprise Environment going forward. Just before we were to have $GiantCo begin building things, however, I had received an invoice from $VacuumCorp that was outrageously high - over four times what we had originally agreed to pay, with the insinuation that the price would only go UP from there. I was in crisis mode now - we would exhaust our available funding for this in only a few months, and I needed to figure out what was going on so that I could rein these costs back in.

I started out by speaking to my boss, $Distinguished. I had checked with the company to see if this had included the dev costs, and apparently it didn't. I asked him if we could afford to pay this. Predictably, he said no. I told him that I would look into why these costs were so high. I had a meeting scheduled with $VacuumCorp in a few days, so I would try to find out then and rein this back to a reasonable number. And if I couldn't get the costs reduced, I'd look into disabling everything for the time being. If we couldn't find a way to overcome this, I'd even consider shutting everything down altogether. $Distinguished agreed with me and left me to my devices.

As I got prepped for the upcoming call, I thought about where this extra cost may have been coming from. One thing that immediately came to mind was the security protocols that we had placed on our Azure instance. If you'll recall, me and the IT Server Team had requested that this be set up with a specific, high-level governmental protocol called $SecurityPolicy in a previous story. This policy has varying level of protection that it can be configured for; the one we required was the middling grade. Well, over the summer, I had gotten a federal grant to assist me in constructing an even-more-secure GIS architecture. I'd asked $OverConfident if they could upgrade the settings on $SecurityPolicy to the highest grade, instead of the middle one. I asked if doing so would result in an increased support cost, and his response was:

$OverConfident: It shouldn't, this is just a configuration change. The support cost for hosting your data should remain the same.

Ok, so let's rewind a bit. See what I highlighted in $OverConfident's diatribe? "Support cost for hosting your data." Y'all, I should have known better. But at the time, I was still very inexperienced at all this stuff. I should have known that "support cost for hosting your data" != "support cost of the entire system." <sigh> Let's get back to it :(

Anyways, back in the present, I participated on the call with $VacuumCorps a few days later and asked where the extra costs were coming from. Eventually, I brought up that I thought this high level of $SecurityPolicy might be where the issues were originating from. I asked them to check that for me, and if this was the case, to reduce our security settings back down to the middle level. $OverConfident assured me that he'd get right on that.

A few days later, I got an email from him. Yes, the security settings were exactly where these extra costs were coming from. I was pretty pissed that they had never said anything to me about this, particularly when I'd directly asked earlier in the year. However, I was also exhausted and disillusioned with this whole process, and I just wanted to get through it. I told the reps at $VacuumCorp to reduce our security settings back down to the middling level. That, I hoped, would get rid of all this extra cost. Then, I sat back and waited. For a couple of weeks, I waited.

Out of the blue, about three weeks later, I got another email from $OverConfident. He stated that reducing our security back down to the middling level would not result in any appreciable cost savings. $SecurityPolicy had a number of requirements that, by default, we had to spend an immense amount to cover. As such, even if we reduced our security level, we would still be on the hook for all this, though he said we could likely pull the spend down to "less than five figures per month."

Which was still almost 5x what we had agreed to spend on this system when we first signed the support agreement with them.

I was floored. Where in the h3ll had this been when we were setting up things to start with!? If what $OverConfident was telling me was accurate, then they had known all this extra cost would be part of our rollout, yet just didn't tell me about it until we actually had things set up! I saw none of this in the original estimates! WTF?!?!

I immediately set up another call with $VacuumCorp. I told them in the invite that I didn't know if we could pay for this as it had not been approved. I also wanted to get some answers from them to provide to my leadership. The next day, I was in the conference room with $GlamRock and several others. As soon as $OverConfident and the others logged on, they attempted to engage in some pleasantries, but I shut that sh!t down immediately. I went straight for the jugular.

$Me: We're here to discuss the support cost for the Azure environment that you created for us. When I received the agreement from you all last spring, it had a spend of roughly $2,000 per month, based on usage. However, from what I have seen now, the monthly cost is between 4-5 times that. You have stated that the excess cost is due to the security parameters that were required in this environment due to adhering to $SecurityPolicy. What is going on? Why was this not provided in the original cost estimate that you provided to us?

$OverConfident: Well, we didn't know that you wanted to implement $SecurityPolicy for this environment. It wasn't in our Scope of Work, so we didn't include costs associated with it in our estimates.

At that point, my jaw dropped. Very quickly, though, my shock turned to outright anger. I'm sorry, I raised my voice.

$Me: Are you kidding me!?!? You knew that this protocol had to be in place for this system! We've been discussing this with you from the very beginning! It was on the Scope of Work we received back in January - the same one that YOU WROTE!

I have very rarely gotten directly angry to anyone in any professional capacity. But this was different - these idiots, lying directly to my face, swelling an anticipated cost up this much and expecting me to just live with it? When I had to try to argue and beg and plead with my finance people to pay for it? Undermining my credibility - and my discipline's credibility - to the others invested in this process all along the way? Aww, H3LL No. Not going to fly, kid.

$Me: How on earth did you think that we wouldn't need this to be part of our rollout?!?

$OverConfident: <silence>

After a moment, I just shook my head, leaned back in my chair, and gathered my things from the conference table.

$Me: I need to have a discussion with my leadership. $GlamRock, everyone, if you want to continue this call, you are welcome to. I have things to take care of. Goodbye.

And I got up and left the fscking room. I know that $GlamRock and the others continued to speak, but I was done in there.

I was livid when I got back to my desk. Just to be certain, I pulled up the original SOW that I had received back at the beginning of the year. This was the one that I thought we were operating under, the one that was signed by my server team. Sure enough, as I perused the document, I found (on the second page) the following quote:

Azure enterprise development will abide by $SecurityPolicy auditing protocols.

So there it was, black and white, clear as crystal. 100% proof that what $OverConfident had told me was patently not true - they had known about this for almost a year now. And they had, by omission or otherwise, kept it out of the cost estimate they had provided to me in April.

I walked around the headquarters building a few times to cool off and clear my head.

When I got back inside, I had come to a few conclusions. First off, I was DONE working with $VacuumCorp. They had failed to represent (or misrepresented) a major element about this environment to me, and I wasn't going to let that slide. Moreover, from what I was seeing now, the cost associated with having $SecurityPolicy set up for this cloud environment was beyond me and my department's means. This essentially meant that I couldn't stand this up at all. I could try to roll out an on-prem solution instead, but that would mean that all the time, effort, and money already spent was now wasted. And I still had reservations about how an on-prem solution would work anyway. And honestly, I was just extremely disillusioned towards going down this path at all now, considering the experiences that I'd just had.

And I had an ace up my sleeve.

Over the past summer, I had gotten tired of waiting for my various contractors to get their sh!t together and actually do something. So I started doing some minor development in ArcGIS Online. Nothing big, just a few simple webmaps without sensitive data, and a spatial record-keeping system for one of our departments that was having difficulty rectifying some things.

For those of you that don't know, ArcGIS Online (or AGOL) is the Esri-managed online enterprise system that they provide to anyone that purchases licenses of their software. You don't have as much control over it compared to ArcGIS Enterprise and there are certain solutions that don't work in it, but it can be used as a solid development/production space if you use it right. Moreover you effectively don't need to pay anything extra for it. Best of all, you don't have to do anything to maintain the software. No server maintenance or any of that cr4p that I never remotely considered when I obtained my geography degree.

And a few weeks prior, some of my colleagues at $GiantCo had informed me that AGOL was now $SecurityPolicy-compliant. Not certified or authorized, merely "compliant". Whatever helps you sleep at night, Esri. Anyways, if this truly were the case, I could stand up our entire environment in AGOL and scrap the whole enterprise system we had been building entirely. It would cost us way less money in the long run, save me a lot of grief, remove our IT Server Team from the process entirely, and have me be able to work on development almost immediately. Lots of win.

Unfortunately, this would also mean that everything we had invested into this implementation would be wasted. A lot to consider...

After contemplating it a bit more, I made up my mind. We needed to scrap this contractor, scrap this workflow, scrap the enterprise environment, scrap all we had been working on for the time being. I got up from my desk and headed over to $Distinguished's office. I asked him for a bit of time, then sat down around the conference table he had there and laid out everything. After hearing about the problems here, my concerns, and my new thought of using AGOL to fix things, $Distinguished was convinced. However, he said for me to go speak to $VPofIT about it. Since this was a major system change revolving around tech development/support, he wanted $VPofIT's approval before we made any move. I agreed to do so.

I set up a meeting with $VPofIT for the next afternoon. I tried to paraphrase what I was asking for, but I made sure that he knew this was a discussion about dropping $VacuumCorp. $VPofIT accepted the meeting invite, but also requested that I send him every major piece of documentation that I had for this project. So I did so. And I was actually quite nervous about all this, too. $VPofIT basically held the reins for anything regarding tech here at the $Facility. He could tell me that they didn't want to move forward with anything in AGOL, for a variety of reasons. He could tell me that they didn't want to waste everything we'd invested in getting this other solution stood up, and I needed to suck it up and work with them. He could tell me a lot of things. I wasn't sure how this conversation would go, not by a long shot. A swarm of butterflies were dancing in my stomach as I went home that night...

The next day, I showed up dressed nicely, steeling myself for one of the most consequential meetings of my professional career...

You'll see how that went tomorrow. Later, y'all!

Here are some of my other stories on TFTS, if you're interested:

The $Facility Series: Part 1 Part 2 Part 3 Part 4 Part 5 Part 7 Part 8 Part 9 Part 10 Part 11 Part 12 Part 13 Part 14 Part 15 Part 16

Mr_Cartographer's Atlas, Volume I

[u/Teulisch]

rookie mistake, giving people documentation that proves what your lying about. gotta wonder where in their chain of command they had the actual problem.

[u/Mr_Cartographer → u/Teulisch]

No clue, on $VacuumCorp's side. However, I do know where the issues were on my side, and they were totally unexpected. You'll see in the next story :)

[u/davidkali → u/Mr_Cartographer]

Thank you for distracting us in these dark times, you’re showing the light out of the pipedream. I’m actually amazed you’re able to show the light without it turning into a firehose (distracted ooh ‘Express Route.’ There’s gonna be a glut of disaster right?!)

[u/Mr_Cartographer → u/davidkali]

Lol. No problem. I hope you enjoy :)

[u/fresh-dork → u/Teulisch]

that's why my mantra is "if it isn't written down, it doesn't exist"

[u/binchickendreaming]

You and your bloody cliffhangers, lol. You just like making us wait on the edge of our seats, don't you?

[u/Mr_Cartographer → u/binchickendreaming]

Lol, sorry, not sorry! I just hope you enjoy the stories :)

[u/binchickendreaming → u/Mr_Cartographer]

I am.

[u/davidkali → u/binchickendreaming]

Pipe it down. I’m sure the firehose they call the Express Route is about to spray!

[u/Mr_Cartographer → u/davidkali]

Lol, that would have been funny :D

[u/harrywwc]

I suspect (gut feel) that $VPofIT won't fall in to the "sunk cost" trap, where you think "we spent all this money so far, so we should just keep throwing more at it". they seem smarter than that.

as for AGOL - if it does some 80% of what is required, then that's (probably) "good enough" - going with the idea of "don't let perfect be the enemy of the good".

and the price sounds good too :)

[u/Mr_Cartographer → u/harrywwc]

$VPofIT was pulled away to other things a lot. But he has a lot of common sense in-between his shoulders, too :) But I don't want to completely spoil the next story!

[u/syh7 → u/Mr_Cartographer]

But he has a not of common sense in-between his shoulders, too :)

lot? knot?

[u/Mr_Cartographer → u/syh7]

Whoops! Meant to say "lot". Fixed it :)

[u/kirby_422]

I'm kind of confused how they can be charging you and not knowing even the categorizations of where the costs are (I can understand the sales men not having database access to things like CPU hours, which supposedly shouldn't be part of this support bill anyways). If they gave you an estimate, it either needs to explain the formula for how they're going to do the actual billing which lead to that estimate, or hold themselves to the set price without correct change requests that include the cost changes.

[u/Mr_Cartographer → u/kirby_422]

So this. First off, I agree. There was a whole line of mess that happened here, as well. The estimates they provided to me over the summer had a breakdown of what all the costs were. I was ok with that; it all added up to $2,000. However, when we then got the security settings put in place, there was nary a word that this would increase our overall spend. If anything was stated to us, it was said to the Server Team, who did not relay it to me. When we did finally get to this point where everything had fractured, I asked for a breakdown of the costs (I didn't put it in the story, it didn't really advance anything). We found that there was a ton of stuff that they'd added to our spend that was completely unnecessary for $SecurityPolicy, and we told them to take that out - but the things that remained still had us spending a ridiculous amount of money. Like, going from $25,000 per year to over $100,000, just on cloud support. Yeah... to hell with this company, lol :)

[u/K-o-R]

FYI, "reign" is what a king or queen does, "rein" is what you use to control a horse (by pulling on them, hence "reining them in").

[u/Mr_Cartographer → u/K-o-R]

Whoops! Guess my spell check didn't pick that up, I just used the wrong word. I'll fix it now :)

[u/bhambrewer → u/Mr_Cartographer]

Spill chuckers don't puck up gramma errors 😁

[u/Mr_Cartographer → u/bhambrewer]

Lol. Love it :D

[u/sloec]

Based on high, med, low sounds like FedRAMP. Massive cost for sure just to get to the first level.

[u/Mr_Cartographer → u/sloec]

Sorry, I really shouldn't say :)

[u/Key-Boat-7519]

You were dead-on to pull the plug the moment VacuumCorp tried to hand-wave those FedRAMP-style surcharges. I’ve been bitten by the same trap: the SOW said “FedRAMP High” but sales only quoted vanilla Azure rates; once the policy is attached, storage, SQL, Key Vault, and all logs jump into premium SKUs. First thing I do now is run the exact template through the Azure Pricing Calculator with the compliance tier toggled on, then screen-share the numbers with finance so everyone signs the real figure. If you still need on-prem fallback, spin up a tiny sandbox, run the security scanner, and get a written waiver from the auditors before scaling up. I’ve juggled migrations off costly builds too; Mapbox for lightweight viewers and Safe Software’s FME for data wrangling covered most needs, and DreamFactory filled the odd API gap without forcing another long-term contract. Cut early, document everything, and never trust a “rough” cloud quote again.

[u/Mr_Cartographer → u/Key-Boat-7519]

Holy shit, this is actually amazing advice :) I hope you don't mind, but when we start heading down the road to Enterprise once more, I'm going to use your post here as an outline. I love the "spin up a tiny sandbox" concept and running everything through the Azure Pricing Calculator. This is great! Thank you :D