I once worked on an in-house ERP system which had been developed over about 15 years by various developers. It was the engine of the entire company, everything passed through it. The CFO and some senior leadership erroneously blamed it for their shortcomings/used it as a scapegoat. When new management took charge, an initiative was started to replace the system with an industry standard solution. Both myself and the CTO (my boss) made it clear that we strongly felt this would not only go way over budget, but ultimately fail as a project.
Having no understanding as to the technicalities involved, the project was given the go ahead by the directors after several meetings with a vendor. After the CTO and I expressed our concerns about the scale of the project and the sheer amount of functionality involved, the vendor gleefully assured us that they were experienced with "migrations of this scale" and were more than prepared, which was music to the ears of the CFO.
Daily 2-3 hour meetings followed (for many months) to define the scope of the project. Within each meeting I sort of zoned out because it became very obvious that no only did the vendor not understand the scale of the work involved, but had started cutting corners everywhere/leaving out crucial functionality, and this was just the scoping stage, no development had even started yet.
I eventually departed the company but kept in contact with the CTO and learned that after 5 years (project was scoped for 2), the migration was abandoned costing multiple millions of dollars with nothing to show for it.
That is surprising, the industry of implementing industry-standard ERP systems is pretty established. I'd have expected it to go way over time and budget (they always do) but eventually succeed. Usually success is in proportion to the company's willingness to admit they're not special and can do the same things as other companies instead of having everything bespoke.
I manage a team of Consultants at a small ERP firm focused on mostly manufacturing and distribution
the #1 cause of failure is summed up greatly by Isaiah Bollinger, paraphrasing "most bad implementations are because people are trying to buck the system they bought, rather than work with it, understand how your ERP, eCommerce or other system does a workflow and match it. There's billions of dollars going in and out of Shopify (or x system) daily, and you are not that special. You will spend 10x as much trying to NOT use the system rather than trying to use it".
I've noticed that the same applies to any large inflexible platform, such as the public clouds.
If you do things the "native" way in Azure or AWS, you'll be fine, just like millions of other customers.
If you try and make the cloud work like your old data centre platform, then you'll have a bad time.
I just watched a customer spend $2M to deploy software routers to replace the "bad" cloud-native routers. Now everything is more difficult, slower, and just all-round bad. But they "had" to do it. (Narrator: No, they didn't.)
The problem is it is often very expensive to adapt other components to fit the inflexible platform.
In your example, the customer may have had software that depended on the routers having some functionality that the cloud native routers didn't. Sure if they had designed for that cloud from the beginning it wouldn't be a big deal. But now, that $2M might be less than the cost of changing all their other systems to work around the limitations of the cloud native router. I've seen situations play out like that a few times.
That kind of logic seems to make sense at first, but just confirms my point: trying to change IPv4 to suit you is a fools errand. Change what you do to hit ordinary bog-standard IPv4 instead and miraculously you’ll have fewer impediments.
You clearly have no idea how complex routing is if that is your take on this.
A random state government department has no "business needs" that require custom IPv4 routing technology that isn't supported by the two biggest public clouds. Any such need is imagined, or an outright error.
In this particular case they were sold a product that serves one purpose: multi-cloud solutions across international boundaries where no single telco can connect all of the data centre locations.
Their handful of locations are all in one city and well-connected by multiple telcos because... they're a state government, not a multi-national corporation. They're blocked by the constitution from expanding inter-state, let alone internationally. That would be a literal act of war.
That didn't stop the vendor's sales team showing slides with titles like: "What if you need to expand into the Chinese market?".
I usually ask, is it a differentiator for our business to run X?
But they don't try to "buck the system" just for the sake of it.
Overcoming process inertia is profoundly expensive and often demoralizing to teams. The project budget for a new system is often pitched as vendor price plus some internal oversight, but this fails to represent the cost of the project exactly because adapting the workflow of a whole division or organization inevitavly costs some multiple of that budget while vendors, consultants, and internal spearheaders all pretend it's negligible.
You're right, ultimately, that failure to adapt is the final damning issue in many of these projects but the root cause of the failure is often that nobody sincerely quantified just how costly and disruptive it will be.
I'm in small and medium business. A lot of homegrown and unique processes that are just dandy. I mean it they're fine. But when faced by things like standard GAAP accounting processes, or even EDI. You can't argue with it. You can't tell walmart oh no well actually I didn't mean to send that 850. It costs you money. You can't willy nilly charge credit cards anymore. Its just life. It makes it hard for them not just in IT but also to hire new people or replace retiring ones. Only Bob knew this process.
What makes the clients we work with great and unique and what I love is the products they make or the problems they solve for customers, but they're not tech companies or banks. The ones who succeed are the ones who focus on thier core value and core skills and not random accounting process X Y or Z.
I work specifically with small and medium businesses too, and can echo that sentiment! I'd love to know more about what you do. Love working with SMBs but don't know a lot of other people in that space.
Its nice. It's certainly not high faluting as a lot of HN jobs but I make good money for where I live, our clients are mostly regional but a lot of the 50 states have customers over the years. Mostly we work on implementing and servicing ERP systems. Our differentiator is our skillet in integrations and holistic problem solving. Since the ERP sits in the middle of almost every IT venn diagram, we run into new tech on a weekly basis.
Besides the ERP consulting bit we sell niche solutions in our product space. Mostly comms(EDI, ecommerce,etc) or payroll/bookkeeping addons.
I worked in consulting for over a decade, and... yeah. When people migrate from one system to another, they try to make the new system work exactly the way the original system does, especially if the original system is homegrown. That lack of flexibility tends to be responsible for more than half the cost (and time span) of the migration.
"We hate everything about our system, get us a new one"
"We want the new system to work exactly like our old one"
It's probably not worth the money...
I haven't pieced those together but of course they go hand in hand.
I had one guy want me to recreate Microsoft word, worts and all, because his current Word wasn't doing it for him
Or refusing to use anything except Office 2003 because newer versions are too different...
This is insane cope. We make technology to assist end users perform the tasks they do. To say "well you're doing the task wrong, the tool is made so you do task X way instead" is to put the cart in front of the horse.
It's not really anyone saying "you're doing the task wrong", it's more like "you're doing it only so slightly differently that we need a few months to write customizations for it. We'll send you the bill".
Those big ERPs are not really blank canvases, but when you need those micro-customizations in the wrong place, they can become one.
End users involved in the integration don't really want to learn new processes or even do things slightly different, as they know that changing process often involves burning a lot of political capital and they often lack awareness to know that "just using the ERP the way it's intended" is cheaper. And consultants are experts in finding a chance to perform those micro changes. It's a perfect marriage.
A cope with what? Thats literally correct. The tool was made to replace 5 assistants, accountants and paper pushers. There's a design to this ginormous system that works best when you lean into it.
Its why picking the right ERP is very iimportat
I understand your point, but that's not how things work. If you source the right ERP you will be presented with procedures and processes that have been streamlined and optimized leveraging previous experiences of literally hundreds of businesses. Smaller and larger than yours. It's an error to dismiss the standard solution without deep consideration. Any process into a good ERP (e.g. managing a deposit payment, managing stock, documents transformation, uniqueness of product codes, and so on) is battle tested and may be ready to solve issues that at the moment your company in not seeing, but perhaps will have to face with the growth of the next five years and in five years perhaps you'll see the reasons why things were setup that way into the ERP. I have been happily humbled more than once by that. Check how things are supposed to work into your ERP, understand them and comply. That's how to have a functioning ERP in your company. Or just don't buy it and go for excel, it's cheaper (but just at the beginning, be warned).
So, basically throw out everything that made them unique and differentiated, and do things like the Germans do.
Not quite.
I never worked in those ERP companies, but a few times I've been on the receiving end, working at companies undergoing a large migration.
It is very often stuff that doesn't really matter, is highly inefficient, and requires small changes everywhere in the system. Death by thousand papercuts.
There is no incentive from both sides to change: the company wants to keep modifying to get $$$, employees don't want to change how they work (because change is often stressful), and the person paying the bills is not getting the full picture.
If there's anything that is actually really "unique" (in a good way), then you spend money. Often this means not customizing the ERP, but actually writing new software that integrates with it.
A wonderful statement straight out of ERP's sales guys (or other folks on the same side dependent on keeping the cash flowing). FYI what we trash here are typical SAP-level migrations and all horrible stories that always come with it, maybe your tiny company does things better but then its a different story in a different market.
I have yet to meet a single company which works like ERP are designed to work. This is their edge over competition, their reason for existence on brutal market. And ERP wipe that out, with the most expensive wiper you can imagine, while selling various bullshit left and right, and constantly lying to given company how everything is fine and under control.
Truly, a way to kill a company. The fact that some survived it all to tell a story just shows how resilient whole such org is to such a massive stressor that ERP migration always is.
They sell first and foremostly a lie - that you can have cover-it-all system just like competition, to match your unique way of working, without suffering tremendously, fit like a glove out-of-box (when reality is exactly opposite). It just never works, more like hammering a concrete glove on your progressively more disfigured hand, while being told how rosy your future will be.
I feel the same above technology in general.
E.g. many professionals jump from technology to technology, rather than mastering how get stuff done with one so they end up being mediocre their whole career.
6 years ago I was called as a contractor by the company I work for as they were desperate that their ERP was just a money drain and despite the money was not functioning at all. Fast forward 6 years. I'm a manager at that company, responsible for business processes and it systems. The ERP works great, everybody's happy. Secret recipe: dismantle custom company processes. Culture change: realize that company's not that special and don't need special ERP recipes. Takeaway: if the ERP standard is set up in a certain way, probably there's a very good reason underneath that setup.
Sales is mostly just lying to collect a commission check.
It's inevitable that Sales will put Product in a bad spot because they're too good at selling their own innocence.
I was a top 0.1% salesperson nationwide in car sales. I read a bunch of lean startup stuff, learned to make a minimum viable product, and then started selling to companies with 20-80 employees. I'm likable and good at selling, so I got 34 companies using this garbage I made. It's the worst nightmare ever to keep people motivated to use it, keep fixing things I made as a rookie developer, keep adding or saying no to features.
Overall the product isn't really needed, and sorta sucks too. If I was an typical developer trying to pitch a startup idea to businesses, it probably would have never got off the ground and nobody would have wasted any time. Maybe eventually the developer would have landed on an idea so good it had REAL PMF, and made that.
But no, instead I sold some garbage and now I'm stuck working on it. There is such a thing as being too good at sales. You don't want sales talking people into bad ideas.
Amazing story. Love your honesty.
Is there a world where you hire someone to build a more maintainable replacement?
> now I'm stuck working on it.
Why not put those sales skills to work and sell the business to someone else?
Hehe, reap what you sow :) But - you are not really stuck though. If you have managed to go from 0 to 34 customers before you can do it again. It is totally an option to drop that product and/or startup, and go with a better one (guided by your learnings). Up to you to decide what is the best way forward :)
Depends on the type of sales. A pretty good indicator is how many times a customer makes a purchase from the same salesperson. If it's just one purchase (like ERP consulting) it could be grifty/etc like in this case. If they're buying from the same guy for years, there's often very different types of salespeople. I used to think all salespeople fit the sketchy used car salesman type, but after working with great salespeople I know better. This is a big blind spot for us techical/engineering types.
When you are working with the same sales people multiple times, it is an instance of treated prisoners dilemma. Used car salesman is just one time prisoners dilemma.
Quite the opposite, they are experienced with those endless migrations.
French poet Paul Valery once said: "A poem is never finished, only abandoned".
It is the same for those projects. You just gotta keep paying forever.
They’re not even lying, usually. They just don’t have the expertise to tell the customer exactly what they can’t have. That’s why you have sales people.
the key thing to understand about ERP systems is that this is their primary purpose. anything else they claim to do is secondary.
Which is why they are vastly better whilst in the middle of a tricky migration, especially if it is going badly, and especially especially if it is being done by the most expensive contractors.
yes, the key is to always have some sort of migration happening so every problem can be addressed by saying "this will be resolved after the new ERP is rolled out"
And you can decline to do literally any piece of work because you need to focus on the migration.
Perfect scheme.
Or “it’s best to wait till the migration complete before our team needs to do anything”
The latter half of your sentence suggests this project may actually have been a roaring success (for the vendor).
Depends on the vendor's goals. If it's to fatten and butcher a pig, and never see any future work from them, sure. If the vendor is hoping to have a sustained profitable relationship then not so much.
Damned. I'd be tempted to write down in my resume "Company would've saved N million dollars and 5 years of their time had they listened to me" were it not such a double edged sword.
No one gets credit for "I told you so" - look at it this way, you didn't have the skill at the time to make the company listen to you.
How could you, in such a case?
They ignored the CTO on an issue that was 100% tech based. There was no winning, the board decided to ignore the person they hired to make that sort of decision for them.
I don't know, and maybe there was no solution. Just offering a different perspective.
short the stock
Better to write "managed critical scoping phase of $xx migration". Not your fault the company priorities later changed and it wasn't completed!
Change this from ERP to POS and you've scripted 9/10ths of the pains for many quick service restaurant chains.
Story time: people are dense as hell. This one idiot put in his personal SSN to open $XX/m a year POS account. This account currently manages just over a hundred locations, and this fool's TIN has just been raking in millions for a few years all because 'I needed to get us signed up!' And what's better!? They still can't get out of the wet paper bag, because otherwise "we'll someone might start selling candy or something unapproved" > we'll isn't that's what your franchise liaisons and agreements are for?! I'm just beside myself.
I'm so hyped on this because this one 'immovable problem' prevents me from having API access to my locations accounts. Everything just hurts. /rant
Would you mind breaking down the situation in a little more detail? Not my field and I find your story interesting.
I also don't totally understand, what's a TIN for example (Taxpayer Identification Number?) and why is it making money? (Don't taxes go to the state?)
Yeah, it’s the identifier the government uses to attribute income for, and hence calculate the resulting taxes.
In this case it sounds like a midsize business is using some random guy’s number, so the tax authorities are under the impression there’s a guy pulling in millions personally, and a company with no income. Given that individuals and companies get taxed differently in many different jurisdictions concurrently this will be a mess to untangle.
Wow that's amazing, thanks for explaining
I swear the only way to get an ERP or POS change to work is to start a competitor using the new solution and wait for them to acquire you.
And then live forever with two systems running side by side.
What happened to the CFO?
Hmmm, maybe check his LinkedIn where he brags about the successful transition and ERP migration he led at that company.
That happened in my company. The CIO left bragging about implementing itil on servicenow. What they did was replacing RT for servicenow, and only for some ticketing queues. Nothing else, no change management, no cmdb, only an incident and non standard requests queues.
128k/ year, while RT still runs in a VM for all the other ticketing queues.
Lots of powerpoints.
probably got hired by the vendor
I read this and immediately became sick to my stomach. I despise working in corporate america.
America might lead the way, but corporate anything, anywhere is pretty terrible.
Consulting:
If you're not part of the solution, be part of the problem.
Or to put it in other terms, the boxer who throws the fight also gets paid very well.
Hershey? https://pemeco.com/a-case-study-on-hersheys-erp-implementati...
The nice thing about consulting at Hershey is that they have candy closets where you can grab what you like, it occasionally includes new demo candy. And the whole town smells like chocolate.
I’m sure it worked out well for the pocketbooks of the new management.
Story as old as time. It's a second system, they rarely work. The only hope is to come up with something small and exciting the a new generation moves too. Then you can build it to become the new thing.
Trying to build a total second system is nearly impossible if it's big and old.
Was SAP or IBM the external vendor? :)
I bet it was Oracle, SAP, or IBM. The hydraheaded horror of enterprise bloated licenses.
I feel this on a deep level since the ERP system at our company is in the middle of moving to one of these big vendors and so much is simply not working out, extending timelines and development cycles. Cost only goes up and ROI is pushed out beyond the horizon.
PLM is equally bad and a horror show of legacy vendors trying to sell their solutions with the promise of customizability. Again the small, more modern players in the PLM space are constantly ignored for the big legacy ones and it turns out those legacy platforms even in their latest iterations are inflexible at best or downright hellish at worst. The reason behind all of this? Service contracts and vendor lock-in are the main drivers of value for these vendors, rather than quality (and modern) engineering.