I’ve just been through the umpteenth difficult technology choice discussion in my career. You know the scene. A team split into two “camps”, each with their own favoured choice of technology. Each camp getting progressively more insular with each passing day of the “debate”. Frosty encounters at the coffee machine cooler.

Its just a crap way for adults to carry on, and it’s not a work environment I enjoy being in.

So this time I jotted down some reflections that I hope I will recall writing the next time I find myself in this situation, so that I can pull them out and offer them as “terms of engagement”. These seem obvious when you read them. But that’s because you’re probably out of the “heat of battle” now as you read them. Team dynamics, ego and the workings of human minds seem to produce this scenario too often for it to be obvious all the time. So it seems we need to remind ourselves over and over.

• Every debated technology choice is a trade off. There is not an objectively right answer. By definition this is the case: if there was a clear-cut “right” choice, there would be no debate. But if there is debate, it’s a trade off. One tool likely appeals to some of the team for characteristics they value highly. The other tool appeals to other members, for characteristics they value, but which are different. Acknowledge this as a team.
• So making a choice in a debated technology decision is a compromise. Call it as such. Instead of calling it a “technology choice” or “stack choice” or “framework choice”, call it a “technology compromise” or “framework compromise”. The language will communicate the reality of what the team is doing, it will emphasise that it’s a team decision, and it will help to de-escalate emotion.
• There should be no sulking following the decision. There should be no “I told you so” when the weaknesses of the chosen technology surface.
• Nobody on your team is stupid. Nobody on your team is “not listening”. They are listening and choosing to offer an alternative view point. And they are plenty smart – just know there are many types of intelligence, and while you might excel in one measure, your team members might excel in others.
• Realise if you chose to dwell in the narrative that “they” don’t get it, it’s likely that “they” might be dwelling in a narrative of you not getting it. Rise up one level of thinking.
• If you can, use both technologies for a trial period. Probably at some stage you will need to make the compromise.
• If the “losing” side cannot bring themselves to work with the technology supported by “winning” side, the “losing” side are welcome to leave the team. This should be mentioned. A successful team has an inner culture. A team of divided mind will deliver suboptimal results, and working “together” will be a pain. So this needs to be surfaced as a discussed reality too. There might be a team next door who made the opposite technology choice, and who are of united-mind and who are successful. That team is a better place for the disgruntled members of your team. An organisation should acknowledge this reality, and support the settlement of people into tribes that are aligned with their mindset.

Here are some essential questions when it comes to delivering software:

• How can I tell when something has gone wrong with my software?
• How can I diagnose what went wrong
• How can I be sure my fix works
• How can I get my fix into the live environment
• How can I tell how much delivery of my system costs

Devops is the process of making the answers to all of these questions be an assured “easily, and quickly”.

So, it means monitoring, alarming and alerting. Of infrastructure, so you know if CPU is exhausted or a disk fails. But also of your app’s behavior – HTTP 5xx error codes, failed delivery of messages, exceptions with particular inputs. Something has to go red when there is a failure.

It means having access to infrastructure and application logs. The ELK stack or Graylog are good and popular examples. To diagnose what went wrong, I want to view the logs around the time frame of the failure, and get a good error message. Writing good error messages is a cornerstone of Devops strategy. Don’t look down on it. The Ops part of you will thank the Dev part of you one day in the future.

It means having tests for your code, and it means investing the effort in making code testable. When it comes to fixing an issue, you write tests to prove the fix. This is not strictly a Devops practice, more an Agile one, but I’m struggling more and more these days to see the difference. And also less inclined to care about the difference. Devops is about caring holistically about continued delivery of service to customers, not about process names.

It means having a continuous integration and delivery process. An automated one. So when you check in your fix, it gets built, it gets tested, packaged, and deployed onto an environment that is identical to production, but maybe on a smaller scale. Thereafter the package is promoted to environments progressively more like production, until it gets to production itself.

It means having tools or reports that tell you how much your infrastructure costs. It used to be called Total Cost of Ownership. Whatever happened to that? Every cloud platform I have worked on so far offers this. I’m paying $5,000 a month.

What’s less quantifiable in many systems, is how much money the system is making the business. Maybe this will have its day – “FinDevOps”, where the finance team are involved in the software delivery as equal partners, and they want to see a taxi-meter style “profit for the day” gauge on Grafana. I’ve seen such a thing once.

Anyway, enough day dreaming.

Devops is also a culture. One of end-to-end caring. Having devs on front line support is a super way to bootstrap this mindset. “If you wrote it, you run it”. Suddenly it gets personal. If a dev realizes he or she is going to be call on Saturday night, suddenly unit testing makes sense! A fast build makes sense. Continuous Delivery makes sense. Monitoring makes sense. Automation makes sense. Retrospectives following an incident make sense.

I have worked in dev environments that were disconnected from delivery and support, and there has been a cynical attitude to “best practices”. Especially retrospectives. Devs can be very cynical about retrospectives. But if you’re going to get a call in the middle of the night on a Saturday to fix your stuff, you want it done fast and reliably so you can get back to bed. Or back to the pub.

And then on Monday morning you will be very motivated to say “Hey folks, we screwed up with that last delivery. How can we do better”. Now you’re very motivated to have a retrospective all of a sudden. And to implement the findings!

So Devops is about caring. Its about not being siloed. It’s not “somebody else’s problem”. It’s a spirit of collective responsibility. And if you go the whole hog and involve the finance Finance people too – give them a taxi meter so they can get excited and join you – you might be starting a new movement.

The 21 Inc Bitcoin Computer is getting a lot of stick. Having a hasher in there is confusing the issue and becoming the focus of people’s view of the computer as an expensive miner. That is unfortunate. But…

21 Inc themselves say the point of this thing is to discover what might be possible when Bitcoin is native to the OS and there is a marketplace to buy and sell products and services. So it’s an exercise in learning. For all of us.

I am sure creative developers around the world will come up with uses for this thing that I cannot imagine, and maybe even 21 Inc cannot imagine – I think thats the result they want.

But even without having to imagine too much, I can see that one use of this device is to charge a tiny amount on a per request basis for all your existing web services. Translation services are what 21 Inc hint at in their launch of the device.

So here are 21 web services that could be easily migrated to a micropayment-per-request model:

1. Watermark a PDF or Jpeg. Charge per page or per watermark.
2. Apply photo effects to an image. If you used photoshop effect names and parameters, photoshop would be at the end of a URL. Nothing to stop Adobe doing this.
3. OCR. Post images, get back text.
4. Audio to text. Post an mp3, get back text.
5. Lookup Foreign Exchange rates. Charge one amount for live prices, another for 15 minute delayed prices, a third price for accurate to the day prices. Similar model for stock quotes.
6. An ad-free search engine, paying 0.00000001 bitcoin per request, say, instead of having to look at advertisements
7. Execute a nominated docker image with specified parameters.
8. Store this file for me.
9. Text to voice. Parameters might include: human or synthesised voice, male/female, accent. I want a human female with British accent to say hello world
10. Snail-mail a pdf to a nominated address.
11. Search 21 Inc services by richer metadata. For example, find me a text to voice service backed by a human that can deliver in 1 hour, that has a rating of 95% and costs 2c a word or less
12. Spamcheck this email.
13. Spell correct this text file.
14. Ad free twitter. Small charge per tweet. I wonder if this would work with Facebook? Certainly would with WhatsApp. Micromessaging paid for by micropayments.
15. Find me 10 images that are copyright free, 600×300 or bigger, and are associated with the word ‘goal’
16. Micropayments to send email. 0.00000001 btc (say) per email or per recipient. Would hurt spammers, and regular users would probably be happy to pay this tiny amount.
17. Send 1 btc to this nominated bank account. Integrate with legacy banking. Might be beyond our civilisation this one.
18. Give me last years accounts for this company as json
19. Give me credit history on this person as json
20. Give me a car insurance quote with these parameters as json
21. Find me sold house prices for the last six months in this post code.

Once you get going with this it gets easier. But like I said. This list was produced by mechanically adding the concept of micropayments to existing webservice ideas. The real fun will start when people start inventing new ideas based on having an api for internet money.

I read someplace recently that our advertising based model of monetisation on the internet was an inevitable consequence of not having internet money when the internet started back in the nineties. And the article dared us to ask when kind of internet we would have now if such a thing existed back then.

For older developers this might be hard to do due to being conditioned to this way of thinking. And it might also be hard for current consumers to envision as they too have been conditioned to expect “free” software. The chap beside me in work told me his dad was reluctant to pay the £8 a month to use Microsoft Office for his work. He thought it was outrageous. That’s the market we have built over the last twenty years.

But the new generation coming through have this opportunity now, to re-envision the internet in this new way. Money does now exist as a content type. Software has progressed from product to software as a service, and now it can move to software as a paid request. It will be exciting to see what happens.

We chatted today in work about generating reports that aggregate many peoples trading positions over many stocks. The way we do it quickly slows down as data grows. So I wondered if we could use some tricks to make a faster reporting system, that also might be more general, and also independent of storage technology.

We can easily enough produce an endpoint that provides a single days trades in json, a bit like this: /trades/22-March-2014 produces

[
	{client-id: joe, 
	 trades:[
		{quantity:101, price:34.5, amount: 3484.5, stock:AAPL},	
		{quantity:50, price:32.65, amount: 1632.5, stock:AAPL},	
		{quantity:-30, price:35.1, amount: -1053,  stock:AAPL}	
	]},
	{client-id: mary, 
	 trades:[
		{quantity:-1000, price:2.78, amount:-2780, stock:BPL}	
	]}
]

What we chatted about is providing another service that hits this url each day and auto-aggregates the data for us. This aggregating system would be configured as follows (using pseudo-json):

{
	client-id: {
		trades.stock: {
			sum: quantity,amount
		}
	} 
}

so now we could go to

/aggregate/trades/22-March-2014

and it would show us the summed position of each client for each stock like this:

[
	{client-id: joe, 
	 trades:[
		{quantity:121, amount: 2685.5, stock:AAPL}	
	]},
	{client-id: mary, 
	 trades:[
		{quantity:-1000, amount:-2780,  stock:BPL}	
	]}
]

What about if we wanted to see an aggregated report of trades between 01-Nov-2012 and 22-March-2014? This aggregating system could also auto-aggregate in blocks of time, so there would be an aggregate for each week of the year, each month of the year, and each year. If we coupled this with another little restful service – lets say

/range?from=01-Nov-2012&to=22-March-2014

which would return how many days, weeks, months and years there are between these dates:

[
	{month:Nov-2012},
	{month:Dec-2012},
	{year:2013},
	{month:Jan-2014},
	{month:Feb-2014},
	{week:1-March-2014},
	{week:2-March-2014},
	{week:3-March-2014},
	{day:22-March-2014}
]

We can now go down through this list, getting the aggregate for each block of time, and aggregating with the previous one – folding the results into each other. All the constituent aggregates are prepared, so its a quick look up for each, and a quick process to aggregate them. It should be possible to make a system like this work for any json data, and it should be able to support several kinds of aggregating functions.

I’m sincerely hoping the demand for this requirement in our system continues, as it would be fun to build something like this. If feels like the right separation of concerns. Of course, there is every chance something like this is already out there – in which case I hope that gets pointed out to me.

Would it be possible to build a system that maintains reputation for a bitcoin wallet address? The reputation system would itself be in the blockchain, so no central authority. Wallet addresses could be claimed by individuals or organisations (no sure how).

Wo when I’m making a payment using my wallet, it could say “The wallet you are transferring to is owned by amazon.com and has a 100% reputation”. Versus “The wallet you are transferring to is unclaimed and has a 23% reputation”. This would make wallets more user friendly and tactile – one of the biggest challenges bitcoin has if its to achieve mass adoption.

you know the way powerful people in movies say “Leave Us” when they want to be alone with somebody they want to give a right good talking too. I want to do that in my next meeting. To somebody talking shite, or being treasonous in some other way. Following the offending statement, I’d go all quiet, fix them with a steely stare and, in a deep and steady voice, say “Leave Us”, keeping the stare going, while the others scuffle out, backwards. Now we’re alone. I imagine the room dimly lit, by candlelight or burning torch ideally, but I suppose the dimmer switch will have to do. Health and Safety. I’d arrange the lighting further so only one side of my face is visible, the other in shadow, eye sockets deepened, maybe a bit of stubble. Nobody messes with somebody framed like that. I suppose at this stage it would be appropriate for me to tut-tut ominously, stand up and step slowly the long way around the table (still in half-shadow) stopping behind the seat of my wide-eyed subject. And pause. Unsheath my company pen, fix it purposefully across their throat, lean in and whisper menacingly…well, whatever it is I want to be clear about. “You WILL come for extra hours on Monday” or “I KNOW you took the last of my pink post-it notes.”, it depends. And now, another episode of Game of Thrones. Leave Us!

I got a moment of clarity today on why I am generally against things like maven and spring.

Our project used to be assembled using a massive Builder class. It was maybe a thousand lines long, certain methods had to be called before other methods, to make sure the relevant objects were created in a proper sequence, and it was hard to follow. Spring advocates asserted that this abomination would be solved by going the Spring route.

Around the same time, our build was becoming unmanageable. Specifically the number of dependencies was getting too large, and too complex to understand. We had jars shared across projects, and ran into divergent needs. Maven advocates asserted that this abomination would be solved by going the maven route.

Both situations have something in common. In the first, the Builder abomination was telling us “your app is too complex, split it up, or simplify it”. In the second, the awful build script with all the dependencies was saying the same thing – “your app is too complex, split it up, or simplify it”.

In both cases we experienced pain, we knew something was hurting. But rather than listen to the pain and try to understand what it was saying to us, we chose to medicate the pain away using tools.

Maven kinda seems to help with dependency management, with declarative and transitive dependencies, but now we have a 50MB WAR file. It contains libraries totally unrelated to what we are doing – like jfreechart and we chart nothing – that come in transitively and are never used. Few people on the team know this, or seem to care. Mentioning that we have such a fat app is met with a shoulder shrug. We prefer to keep away from the pain.

Similarly, now that we’re on Spring, there is no single horrible Builder class that you swear at every time you have to change it. Instead there are many smaller xml files, and autowiring and annotations that make the wire-up happen. The organisation has invested in an artefact repository with people looking after it etc. All these smaller parts and activities seem to feel less painful. But I think the sum of pain is at least the same, the complexity is at least the same. But it all has the seductive quality of being less in our face.

So as I sat there today for several minutes watching maven download jars, I realised I want the pain that its shielding me from back in my face. Don’t medicate me away from pain with these abstractions. In the human body pain is feedback calling attention to something that needs to be fixed. The wise response is to pay attention, not to medicate. So this is why I am against maven and spring and the like. They attempt to cover over things that I want direct contact with, things that I want to feel, things that give me feedback. If my app is hard to configure, I want the feedback. If my app is a 50MB war with a ton of dependencies, I want the feedback.

So I’d prefer to strip these things out and get more down to the metal. It would be painful, certainly, but I’d welcome that. The app would be the better for it.

“I think we should use Spring.”
“Over my dead body…”

“Maven is the only way to go.”
“Maven makes me want to hurt people…”

“I think we should have simple data objects and put all the logic in services.”
“No way, Eric Evans is a god and you need to read his bible…”

I’ve seen, heard and been involved in too many of these “discussions” recently. They’re wars really. I’ve not seen one come to a satisfying close. At best the warring parties walk away from each other without coming to blows, grumbling about how stupid is the other, and go on with whatever approach they happen to prefer anyway – the “right” way.

The damage from these kind of wars can be contained if they are across teams, but when they are between team members it can destroy the team. If half the team wants Spring, and the other half don’t, it can make for some very awkward pairing, “code wars”, terrible morale, and impaired productivity. And a loss of that very precious life experience – happiness.

For a long time this bothered me a lot, and I banged my head trying to find a solution. Surely there is some way “we can all just get along”? Well, recently I’ve stopped. I saw that there’s no point. There is no solution. Because there is no problem. Some people like Windows, some like Linux, some like OS X. Is there a solution to this? No, because its not even a problem. Its a blessing. A Linux die-hard could be a Linux die-hard for life, and good luck to them. Some people prefer BMW, some swear by Mercedes Benz, others by Skoda. Some people are Hindu, some are Protestant, others Muslim. Good luck to the lot of them.

Therefore, coming back to software development, its more important, in my mind, to be honest about a team’s tooling and development culture, and to hire members that fit.  If a team is all about Spring, there is no point hiring a developer who openly states that they hate it. If a team is all about domain driven design, there is no point hiring a developer that states that separating data objects and service objects is the only way to go.

And for you as an individual developer, you need to be honest with yourself and with others about what your instincts are, and find teams in which you’re a fit, rather than an annoyance. And if you do hate Spring (presumably with reason), be honest enough to say it. Have the courage of your convictions. There can be a fear accompanying this kind of personal honesty that makes you think “damn, if I say I hate Spring, maybe I won’t get this job I’m interviewing for. Or maybe they will think I am close-minded.” Well maybe you won’t get the job and maybe they will think you’re close-minded. As long as you know your reasons for hating Spring, then you can rest assured that you have made a personal, informed choice. As for not getting the job, well – phew! – you’ve saved yourself and the interviewing company much unhappiness, and you stand a chance of finding a team on which you’ll flourish.

One qualification that is essential to add, and which keeps open the door to development and learning, is this: any individual or team that finds themselves aligned to a particular approach must still be open to listening respectfully to advocates of other approaches, and maybe even being friends with them 🙂 Keep questioning and reading and studying and talking to people and trying things out, and stay honest with yourself, because at the end of the way, maturing as a software developer is never about becoming finally right, but by becoming increasingly less wrong.