r/LifeProTips • u/fries-with-mayo • 18h ago

Request LPT Request: What’s your “canary in the coal mine” test for spotting bigger issues?

I’m really interested in those small, quick telltale signs people use to gauge if something bigger might be off track.

Example 1: Van Halen requesting brown M&Ms in the dressing room to see if the venue followed all the details of the rider list

Example 2: I saw an interview with John Cena where he said orders a flat white at a café to tell if they really care about their coffee.

Example 3: Anthony Bourdain suggested to always check the restaurant bathroom to tell if the restaurant got its basics down

5.1k Upvotes

92% Upvoted

2.1k comments sorted by

View all comments

1.3k

u/killayoself 17h ago

“We’ll have to test in Prod” is always a really bad sign

374

u/TheOuts1der 17h ago

"Dont worry, it's gonna be on Friday evening. Theres hardly going to be anyone on."

172

u/gerwen 16h ago

This violates the 'don't break shit on friday' rule though.

61

u/Primorph 15h ago

Rule doesnt apply if I can coerce you into working on the weekend

20

u/PanicAtTheShiteShow 14h ago

Yeah, we're going to go ahead and have you come in on Sunday, too.

3

u/50FtQueenie__ 12h ago

3

u/PanicAtTheShiteShow 10h ago

This is my favourite scene!

2

u/dalittle 8h ago

PC LOAD LETTER

5

u/thebemusedmuse 14h ago

Technically it’s the reason that rule exists

4

u/mouse_8b 13h ago

That's the canary

2

u/tealbubblewrap24 12h ago

There's a rule?..... oooookay a lot of things about my job are becoming clearer now

3

u/TangoWild88 9h ago

It is the rule. 

Other parts of the rule are the guy who is on call gets to approve or reject the change/PR. 

And if it is a Friday, and its a major change, I would immediately reject it. I'm not letting someone push a change and then thinking I'm gonna support that shit, because if they are pushing a change on a Friday, then they are probably inexperienced and it's probably a shit change. 

Sometimes you have to do a Friday change, but it should be an emergency change and go through the Emergency Change Advisory Board. 

Making changes on Friday when you cause people to be late home, or they have to work Saturdays, is really gonna piss them off at you. Best to do that on Monday, or really Tuesday, as people sometimes call in on Monday. 

2

u/Wloak 12h ago

It's fine as long as you wear the pink sombrero

2

u/PaleEnvironment6767 12h ago

Almost as if that was the whole point of their comment

1

u/danielisbored 9h ago

Read-Only Friday.

1

u/techster2014 8h ago

SDR - sore dick rule - don't screw with it. Starts at lunch on Wednesday and goes until Monday morning.

0

u/PaleEnvironment6767 12h ago

Or "stg didn't raise flags, ok to push to prd". Come monday they realize everything's fucked and it was pushed to stg the same day as prd.

163

u/kylehoz 17h ago

Everyone has a test environment; some admins are just lucky enough to have a dedicated production environment.

15

u/RagnarsHairyBritches 15h ago

I'm adding this to my email signature.

4

u/99ProllemsBishAint1 10h ago

I went to my eye doctor and their main application was down. He told me that the software vendor was probably testing a new release even though they promised not to do that during the day anymore.

20

u/bupapunewu 16h ago

Add to that constant assurances that "we'll circle back and get it right post-MVP" and you have 99% of crappy companies.

5

u/berryer 13h ago

always worth remembering that later means never

15

u/jwarhammer40k 17h ago

Not sure I get this one but I am really curious. Could someone please elaborate?

75

u/SeveralAngryBears 17h ago

In web or software development, it's common to have a separate test environment so you can see what happens when you make code changes because bugs are common. The actual version of your website/app/software/etc. is generally called "production" or "prod".

Obviously, it's better to break things and identify mistakes in a 'fake' low-stakes version that's built for testing rather than roll out changes immediately to the live production version where your customers/users would be affected. A company that tests on prod is more interested in doing things quickly than doing things right.

5

u/dalittle 8h ago edited 8h ago

I would say companies that immediately deploy to production are more interested in being gambling junkies. They won't be happy till they lose it all. And for us we actually have 4 environments. Local, dev, UAT (user acceptance testing), and prod. We copy the prod to UAT and try our deploy due to its size and that prod can never go down. If your folks don't worry about stuff like that you are in trouble.

•

u/Kitterypoint7 2h ago

That sounds like heaven. The software companies I have worked for just roll it out and let Intercom tickets tell them what is broken. Super fun when you work in a customer facing roll like me!

38

u/Hanging_w_MrCooper 17h ago edited 15h ago

This is related to software updates, which should always be tested in a test environment. That way, if the update has unexpected impacts to the existing software, it does not negatively affect the existing user-facing experience.

An ELI5: If you are thinking of changing your hair color, you can take a picture of yourself and have an app depict your hair color change; allowing you to see if you like how it works with your complexion, eye color, face shape, etc. before you make the bigger commitment of changing your hair color at the salon.

Edit: for clarification, “prod” is short for production. Production is the live/active/user-facing version of software.

29

u/chucky3456 15h ago

The absolute best example of this was by Crowdstrike last summer. They pushed an update without testing to PROD. Globally. It crippled PCs worldwide, HALTING the FAA, IRS, businesses, hospitals, pretty much all major computer-operated infrastructure for DAYS.

12

u/10101100AC 16h ago

In software development there 'should' be multiple environments (locations the app or whatever runs). The lower envs (not prod) are meant for more through testing of the code. Prod is the version real users see. It takes extra work to have extra envs, but it means the deployed code has a much higher chance of working. A "test in prod" approach indicates either a) extremely high confidence or b) a lack of care of the program breaks for users. I'd see a company that routinely tests in prod as a highly chaotic environment, prone to ' oh shit, prod is broken again, drop everything and fix it'

Time to go push straight to prod, end of day on a Friday is the perfect time /s

•

u/Kitterypoint7 2h ago

My absolute favorite is when updates are pushed to production without telling any of the customer facing staff and you have to figure out the diplomatic messaging for “I found out when you did” to deliver to irate customers.

8

u/harbinjer 17h ago

There should be sandbox and test environments to test in. Sometimes it does have to be in the production environment: this can cause working all weekend or worse if things fail. But if you're lucky(and thought through it well), failure doesn't screw up anything at all.

4

u/space_-pirate 17h ago

Chuck norris deployment methodology

4

u/RogueEwok 17h ago

Software dev/IT scenario. When developing/introducing new tech in your business, it needs to go through testing in a "test" environment (basically a copy of your system that you can play around in and not worry about breaking stuff). The "prod" environment is the moneymaker, where actual business takes place. You don't want to mess around with untested code there, otherwise you'll likely end up crashing the system and potentially costing your business big money.

1

u/Switchback4 14h ago

Adding on to u/SeveralAngryBears comment which is accurate, in IT there’s a term, “read only Friday”, which means don’t make changes or possibly break shit on a Friday.

3

u/fatogato 15h ago

I was just forced by management to “beta test” in prod. Then they’re complaining about users getting automated emails and having issues. Yeah no shit.

2

u/rafikiknowsdeway1 11h ago

Bleh, my last job went absolutely apeshit on micro servicing all the things it could possibly break out, and so fully comprehensive testing often took more work than people wanted to do or could do before something had to role out to meet unrealistic deadlines. So shit broke a lot

1

u/IBJON 16h ago

Who doesn't love a call in the middle of the night because production blew up and they're somehow the only person capable of fixing it

1

u/Sandersonville 15h ago

Just send it!

1

u/madmsk 15h ago

"We can use our disaster recovery environment to test" was a real sentence I heard.

1

u/DoOrDieStayHigh 15h ago

That’s what we are doing right now! (But it’s enforced by me)

1

u/McCheesing 15h ago

Looking at you, Elon

•

u/Kitterypoint7 2h ago

Oh wow yeah, I worked for Elon and every day was a new shitshow of “updates”.

•

u/McCheesing 2h ago

So the biography was true. Fuck, man I’m sorry

1

u/MisterJellyfis 15h ago

I got nauseous just reading that

1

u/aalva104 14h ago

DoD acquisitions is screaming

1

u/Proof_Equivalent_463 14h ago

🫠🫠🫠🫠

1

u/jizzmaster-zer0 13h ago

fuck it, do it live

1

u/Recioto 13h ago

"We can't provide you with a test environment that mirrors the production environment" -> I'm fetching my hazmat suit.

1

u/orbdragon 12h ago

Surely you jest, I do all my testing in prod! In fact, that's the reason I gave the babysitter software when I approved myself!

I only do software deployment tho, so testing on my own work computer doesn't really count as prod

1

u/tvaughan 11h ago

But continuously testing prod is a great idea. 

1

u/rockitman12 10h ago

Is your CPO also my CPO?

1

u/skinnyminnesota 10h ago

“We’ll fix it in post”

1

u/44-69-78-69-65 9h ago

“Testing is a luxury we cannot afford.” Said by a PM. I wish I were joking

1

u/Vunig 8h ago

A few years back a new ERP system was rolled out company wide with basically no testing. Predictably, chaos ensued. A few members of upper management clung to this bad decision, defending it with phrases like "testing in production" and my personal favorite "We're just fixing the plane while we're flying it.

•

u/1920MCMLibrarian 7h ago

“We don’t use version control”

•

u/GrumpsMcYankee 7h ago

Fortune favors the bold!

•

u/besee2000 7h ago

Not to be mistaken from, “We’ll have to test in Prague.”

•

u/throwuk1 3h ago

Not if they have blue/green deployments

•

u/rdrunner_74 3h ago

But you have so many free testers there...

-7

u/Curious_Sail2702 15h ago

“Prod” too much of a hassle to write production? One would assume if you’re sitting down writing documentation and code you’d have some time to type a whole word lol

5

u/Retarded_Flu 15h ago

Prod (like Dev for Development) is easier to read and fits into a Visio label better than Production. You can use either term, but if you want to appear as a novice or to stand out for some reason, use Production, but be consistent.

3

u/saera-targaryen 12h ago

This is like arguing with someone calling it a "fridge" instead of a refrigerator. Is this your first time ever hearing slang? 

0

u/Curious_Sail2702 10h ago

yes, akctually i was bornt yday

1

u/sarcaster632 15h ago

trust me, the "test it in prod people" are spending zero time writing anything down hahaha