Notation</th>	Applied rule</th>	Description</th></tr></thead>
1.0</td>	x ≥ 1.0</td>	Minimum version, inclusive</td></tr>
...</td>	</td>	</td></tr> </tbody></table> Source: NuGet version range definitions</a></p> It doesn't even define whether it will upgrade major, minor, or just patch releases. Horror-show.</p> lock files</h2> Lock files don't actually help much here, but are a good idea, so if you want to use them here's what I've learned about `packages.lock.json</code> files in Microsoft-land.</p>` NuGet is veeeeeeeeeeery late to the lockfile game. A lockfile is just a second file that lists the dependency tree that was actually calculated for a given set of top level dependencies. Useful for repeatable builds and lets you see what's being resolved, but doesn't actually help fix the problem here as the point of lockfiles isn't to manually futz with them, it's just to do repeatable builds. And futzing with them would confuse everyone, and is made a little harder with the inclusion of checksums.</p> `It is possible to` `manually enable lockfiles</a> by adding the following to a csproj file:</p>` <</span>PropertyGroup</span>></span></span> <</span>RestorePackagesWithLockFile</span>></span>true</span></</span>RestorePackagesWithLockFile</span>></span></span> </</span>PropertyGroup</span>></span></span></code></pre> I very much enjoyed this multi-comment rant at the NuGet team</a> for basically being behind the times and always making poor decisions.</p> There is a slightly harder problem that microsoft have so far failed to sort out of managing packages across a whole solution</a> (i.e. multiple projects). Though I can't help thinking that it's mostly hard because of the mess they've allowed to build up so far. (For example encouraging people to create multiple solutions with a mix of project files included, and the horror that is the one-giant-tree TFS system.)</p> Forcing the upgrade</h2> So it seems the only way to force the upgrade of the vulnerable "transient" dependency (i.e. one not directly specified in your .csproj</code> file) is to add it as an explicit dependency directly to the .csproj</code> PackageReference</code> entries. This is lame, and likely to cause confusion later. Make sure your commit message is clear as to why you've added this seemingly nonsense dependency.</p> <</span>ItemGroup</span>></span></span> <</span>PackageReference</span> Include</span>=</span>"</span>System.Text.Encodings.Web </span>"</span> Version</span>=</span>"</span>4.5.1</span>"</span>/></span></span> </</span>ItemGroup</span>></span></span></code></pre>Getting notified of dependency updates</h2> If you run any production code you can run an analysis tool like Snyk.io</a> or dependabot</a> to check for such CVEs popping up in your dependency graph.</p> You'll probably want to install a local version of whatever you use to avoid the CI loop for every change you make, snyk has a visual studio extension</a>.</p> Phone setup notes 2021-12-17T00:00:00+00:00 About</h2> Well this makes a good counterpart to the laptop setup post</a>. Similar idea, it's just a place for me to keep track of all the things involved in building a new phone from scratch. This one also explains to people who look at me like I'm mad why I don't just "do what everyone else does".</p> Why don't you just...?</h2> Most people don't think their phone install deserves a blog post, or even a lot of thought beyond a visit to the Apple "genius" bar (lol) so they can push the "copy everything to my new phone button". I'm not most people. I've been running Linux since approximately Ubuntu 5.04 Hoary Hedgehog</a> in 2005 (the brown one). Even while being a ".NET developer", back when Microsoft were still trying to simultaneously ignore and destroy Linux</a>. What strange times we live in now with dotnet being open source and cross-platform, has the leopard changed its spots?</p> There are many factors in this battle for the phone in your pocket.</p> The GPL's fundamental truth is that there is a tension for control over your devices between the creators of software and the users of that software. iOS and Android both believe that the users can't be trusted with full control over their device because "security". I believe this balance of power is important. Too far in either direction and things start to go badly: All the power in the hands of vendors at the expense of users results in exploitation, unfair pricing, anti-competitive practices, monopolies, duopolies (google+apple), cartels, and lack of innovation and quality due to lack of competition.</li> On the flip side all the power in the hands of "users" (or at least user-developers) as per the pure GPL and it becomes difficult or impossible for capitalist processes to fund innovation and engineering effort due to the inability to capture generated value.</li> </ul> </li> The apps matter, I do actually want to use a phone for useful things, and increasingly web applications are a poor cousin to their equivalent apps. (Facebook, horror that it is, oscillates between making a browser unusable and making it just about possible to read and interact, they really</em> want all that juicy extra data that you can get from being an installed app).</li> </ul> iOS and Android both believe that users can't be trusted with full control over their device</p> </blockquote> This time in smartphones is a moment like my first linux install around the year 2006. I hated the main operating system (Windows) that I was locked in to. All the app[lication]s that I relied on (outlook, visual studio etc) were locked to the operating system I hated. And now in 2021 all the proprietary apps on phones are locked to android / iOS (gmail, authenticator, waze, google maps, banking apps, whatsapp etc.)</p> I could just install OpenGApps or similar to get the google play store on top of a slightly more free android variant; then install all the proprietary apps again, or I could see just how much I can do without google services by using microG and open source alternative apps like k9mail and maps.me. I've already started this journey by trying out many free alternatives and think it's worth pushing ahead. Running a build with no google services will be a good acid-test of how far I've got even if I have to go running back to the proprietary google services.</p> In my switch to Linux from Windows I just tried to do as much as I could with free software such as the gimp and thunderbird, often still installed on Windows, and then bit by bit I needed to fire up Windows less and less often as I found alternative apps and ways of doing things.</p> Any progress I can make here will make the jump to a fully free phone OS less painful in the future, and I'm increasingly confident that day will come just like I don't think twice about running Linux Mint even for work these days.</p> Nursing Homes and Neighbourhoods</h3> I highly recommend reading this article - Purism, The Future of Computers: The Neighborhood and The Nursing Home</a>. It wonderfully illustrates the binary choice between the apple/google duopoly and the DIY open source and privacy movement with the rather apt analogy of nursing homes (apple etc where your needs are met but you give up control of your environment) and neighbourhoods (open source where it's up to you not let the wrong people in to renovate your house).</p> Why not just Apple iOS?</h3> Yes I know the iPhone crowd can just push a button on the new phone and have it look like the old phone, but iOS is the polar opposite of what I want my technology to be: under my</em> control. That means root. That means open source. That means GPL.</p> iOS certainly does have some unique advantages that I will be sad to miss out on but not sad enough to give up all the other things I care about.</p> iOS, the good</h4> A secure boot chain, remarkably hacker-proof - even with physical device access which is very impressive.</li> Impressively effective device-to-device migration of user settings and data (android failed miserably at this in my experiments)</li> </ul> iOS, the dubious-to-bad</h4> Complete reliance on a single vendor for looking after all your precious "data". This in my view is risky no matter how good the vendor is. In 2012 this was demonstrated as Mat Honan had all</em> his data and devices wiped and locked remotely when a hacker took over his Apple account</a>, and he had no other copy of all his precious family photos. A cautionary tale for all of us.</li> Someone I know lost everything on their iPhone when the OS upgrade process somehow managed to do the following: backup the phone to the iCloud</li> wipe the phone and install the updated iOS (I understand this is normal procedure)</li> back up the blank phone to iCloud overwriting</em> the only backup (apparently there was only a single slot)</li> restore the blank backup to the updated phone</li> declare success, leaving my friend with all his data gone and a factory-installed OS</li> </ol> </li> </ul> </li> No root (a problem for syncthing, independent backups and general user-control)</li> No user-replaceable OS (aka ROM)</li> No open source</li> All the apps cost more!</li> </ul> Why not just stock android?</h3> "So you must be an android dude then" you say. Well no because frankly android isn't much better these days. Ever heard of AOSP? (The Android Open Source Project)</a> Well that's an ever-shrinking piece of what people call Android these days. Piece by piece google have been replacing open-source android with proprietary rewrites</a>. Combine that with proprietary drivers for the endless churn in hardware. If you can even install ASOP on a device good luck getting much to work. The idea of android being "the open source one" in the true spirit of the GPL is a distant and fading memory.</p> There is a cartel of industry players called the Open Handset Alliance</a> http://www.openhandsetalliance.com/index.html</a>. These industry groups can be good but I think in this case they are a barrier to innovation and openness.</p> "OHA members are contractually forbidden from producing devices that are based on competing forks of Android" ~ Wikipedia on OHA</a></p> </blockquote> Ars Technica: "Google’s iron grip on Android: Controlling open source by any means necessary"</a></p> Root and backups</h4> One of the big drivers I've had for getting a non-standard phone setup is that I don't want to rely on on google to backup everything on my phone in case it dies, but because Android is by default locked in by secure boot and doesn't give the user any root access or ability to get root access that's a bit of a problem for the normal method of backing up a device.</p> A laptop backup is a question of backing up /home</code> plus a dotmatrix/bootstrap</a> setup for regaining interesting configuration etc.</p> Android is a bit more complex because it has a couple of storage areas:</p> an [emulated] 'sdcard' this is kinda like /home</code> on a laptop and has things like downloaded files; some of the apps use this to save things to</li> storage for every individual app (I learned about the existence of this by losing all my app data)</li> </ul> Every app on android runs under a separate userId, which allows Android to ensure that apps can't read each other's data. This is a good thing for security, but that means if you install a "backup app" it can't read any of the data you actually want backed up. And it can't get root because that's not available to you as a user or to any non-system apps.</p> So that means to backup independently of google, you need root, and to get root you need to unlock the whole boot chain.</p> I did have root on my OnePlus 5t, but an operating system update helpfully removed that for me... breaking my backups... and root access couldn't be restored without wiping the phone... including the data I didn't have backed up because it had broken my backups. Gah. Off the back of that experience I want to make sure that I'm not set up to fight some bloody vendor that doesn't believe I should have root.</p> Any hope of an effective root is dwindling. According to https://www.androidpolice.com/a-new-method-for-hiding-root-in-magisk-may-soon-emerge/</a> the developer of Magisk Hide that prevented root being detected and thus allowed shitty apps to continue to be run has been hired by google</em>. That's google following the advice of "keep friends close, keep your enemies closer" close if ever I've seen it. No better way to shut the project down than to pay off the developer with a "salary". Cough.</p> Something other than iOS or Android and its derivatives?</h3> It's all about the apps.</p> An operating system is not that useful on its own.</p> </blockquote> There's a reason Ballmer shouted "developers developers developers"</a> (amusing as this meme has become in the abstract). An operating system is not that useful on its own. Operating systems live and die by the applications (aka apps) available on them, and the utility and value they in turn bring to their users.</p> iOS and Android have utterly captured the app market on phones, creating a duopoly in the phone space.</p> Even the mighty Nokia (with Symbian</a>), Research In Motion (RIM) with Blackberry</a></code> and Microsoft with Windows Phone</a> failed to win against them and eventually threw in the towel.</p> </code> Blackberry seems to have all but fallen off the radar, they started shipping android in the end basically being a android phone with a physical keyboard thus becoming part of the apple-google duopoloy..</p> So for me that rules out for the time being all the Linux based phones such as a Librem 5 running PureOS</a>. (See below).</p> It's only when alternatives like Librem/PureOS gain traction that we'll see the makers of apps, websites and phones have to create and adopt open standards that allow further diversity and innovation.</p> </blockquote> The fact the Librem exists and seems to be sustainable are encouraging signs for the breaking of the google+apple duopoly on mobiles. It's only when alternatives like Librem/PureOS gain traction that we'll see the makers of apps, websites and phones have to create and adopt open standards that allow further diversity and innovation. Currently we are seeing a repeat of the "works best in Netscape / Internet Explorer" duopoly of the browser wars that was eventually broken forever by Firefox, Opera and friends forcing websites to stop using proprietary extensions and stick to open standards.</p> Dearly departed</h4> Nokia / Symbian</h5> Symbian was briefly ahead on features as an OS in my opinion. I had a Sony Ericsson P800</a> and P910</a> and in their day they were almost</em> amazing phones, hampered by lacking hardware (2g at the dawn of 3g, serial port connectivity emulated over usb, and for god knows what reason Sony were still trying to beat SD cards with memory stick</a>). Consumers flocked away from them in droves to the new iPhone.</p> Microsoft / Windows Phone OS</h5> Microsoft's "Windows Phone" was a triumph of a phone operating system</a>, and everyone I knew that had one absolutely loved the operating system. But Microsoft just couldn't persuade the big-name mobile-app developers</a> (i.e. companies) to invest the endless effort required to create and maintain a third version of their apps for another app store.</p> OpenMoko / Neo FreeRunner</h5> I had high hopes for the truly open Neo running OpenMoko, but it didn't stand a hope in hell amongst the rapidly evolving tech and software at the time. I bought the Neo FreeRunner</a> and it was an exciting time, but it quickly became a relic.</p> CyanogenMod</h5> After many years of being the goto android-variant mobile OS for non-conformists, sadly CyanogenMod</a> imploded in 2016. The code lives on in LineageOS (see below).</p> Promising futures</h4> Two currently active non-android physical phone projects, and several more ongoing Linux based (i.e. non-android) mobile-friendly operating system efforts. Although these will never cross the radar of most people just like Linux didn't for many years) these are important foundational projects.</p> Purism's Librem 5 phone hardware running PureOS (Linux)</h5> Librem 5 phone</a></li> PureOS</a></li> </ul> The Librem 5</a> is worth a look if you want to take it to the next level of openness, however it comes with a compromise in specifications, a hefty price tag (inevitable due to lower production volumes) and no android / iOS apps.</p> You might however want to consider spending your dollars as a vote for the future you want to see.</p> The Librem 5 runs Linux (PureOS), not Android</a>. This is great but as of 2021 does mean it suffers from the "app" problem (see above).</p> Because of the "app" problem I'm not ready to make the jump to a Linux based smartphone that can't run android apps. I'm also not particularly keen to drop quite this much in specification for the benefit.</p> Personally I'm happy with the OnePlus compromise of specification plus apps versus openness for now, but I'm very glad to see Purism pushing the frontier of openness and maybe I'll join their users one day.</p> PinePhone Pro phone hardware (Linux, for now)</h5> The PinePhone Pro</a> is a low spec but very open Linux phone. A follow up to the original PinePhone.</p> "pre-orders from developers on October 15, 2021, and expect to have them delivered by December"</p> Phones like this are important in building momentum in the capability of non iOS/Android software even though they will never be mainstream phones themselves.</p> This is a Linux phone, though in theory GloDroid</a> will provide a build of Android (AOSP) for it, but don't hold your breath. - OmgUbuntu article on PinePhone+GloDroid</a>.</p> UBPorts Ubuntu Phone (Linux)</h5> Canonical killed the Ubuntu Phone and the Ubuntu Touch</a> operating system, but the software lives on in ubports</a>. (Presumably that's short for UBuntu Ports)</p> Plasma (Linux KDE for phones)</h5> Plasma Mobile</a> is a variant of the KDE Plamsa desktop specifically for mobiles.</p> Learn more about Plasma on Wikipedia</a></p> Mobian - (Debian Linux for phones)</h5> Mobian</a> is encouraging to see. Debian Linux's strenth has always been its governance and stability. Debian has been a rock in the stormy seas of open source even as different people have passed through its team. This is because debian has a strong set of organisational rules that are above and beyond any individual contributor. The fact that they are working on a mobile variant is very encouraging.</p> PostmarketOS (Linux)</h5> PostmarketOS</a> is an Alpine Linux (i.e. very small) variation designed to breath life into otherwise obsolete hardware. I expect this will become more useful as phone hardware stops becoming utterly obsolete due to major changes in basic expectations and environment.</p> Secure boot</h2> There's the secure boot chain and device locking. This can be a good thing or a bad thing depending on the power dynamic between users and providers (and maybe even the state/government).</p> It is true that your device is less "secure" with the bootloader unlocked (as the phone likes to scream at you every time you boot it up), at least to attackers with physical access to your device, however I think that's worth the trade for full control of your device (that you paid for and "own").</p> Security is not an absolute</a>. You have to consider cost, inconvenience, human behaviour, the value of an attack target, the motivations of different actors (state, organised crime, hackers, opportunist thieves, trolling friends etc). For me an encrypted data partition and a passcode will do for physical security. In theory you can get full secure boot chain for custom operating systems but even on a Linux laptop this is non-trivial currently.</p> There's an important difference here between google/apple considering the security of their entire device eco-system in the hands of every kind of user, and the security of an individual's device customised to suit them. If google turn off secure boot for everyone then it makes it that much more likely that someone will successfully develop and spread an android virus that corrupts the boot chain. If I turn it of it's a rounding error on the probability, and a marginal difference to the "security" of my own device.</p> Apple's secure boot chain</h3> Apple don't want you to fiddle with the OS they control, so you don't get to touch it without "jailbreaking" it. Jailbreaking is done by exploiting a security flaw, and it's a cat and mouse game with Apple fixing bugs to close holes and hackers finding new ones to regain control of the devices. Not a game I want to play just to have control over my own hardware. This 100% rules out iOS for me, it's a complete dealbreaker. No matter how shiny, how bug free, how capable it is I wouldn't trade unless there was absolutely no alternative.</p> Android's secure boot chain</h3> Android also secure their boot chain by default, however whether you can unlock it and flash a different recovery/bootloader/OS/ROM depends on the phone manufacturer, individual device, and sometimes the phone network provider (aka locking/unlocking).</p> This means choose your device and provider carefully if you wish to unlock without wasting your time trying to jailbreaking a locked device.</p> Artificial app limitations when unlocked</h3> Android has a system that allows apps to hide or limit functionality called SafetyNet</a>. For apps like NetFlix that won't even show in the play store for unlocked phones we're into the mega-discussion that is DRM</a>. In short if a content creator wants to lock their device to a locked-down platform that's fine be me but I won't be using those platforms for my main computing, instead I might have them as throw-away additional appliances like the google chromecast, or samsung TV's built in junk. What really matters here is what alternatives you have available to you. For google pay, I'm very slightly sad to miss out but my contactless debit card still works fine so I'll live.</p> Hardware choices</h2> Given the "app" problem, any devices not capable of running Android or a derivative are currently out of the running (sad times).</p> A non-negotiable for me is manufacturer support for replacing the bootloader and everything above that (OS, apps). This ensures the life of the device beyond the point at which the manufacturer loses interest or does something horrific/stupid. This is also a signal for who is truly in charge of the device once it's "yours" - you the user or the supplier and their developers. By locking you out of making modifications it's a clear signal that you are not to be trusted with the power over your own device.</p> As the useful innovation in hardware slows down (2G to 3G was a big jump, but 8-megapixels camera to 32-megapixel cameras is less foundational) it will be less often that devices will become obsolete through huge leaps in capability. It will also mean that phone hardware will inevitably become more commodity and less proprietary, giving a chance for more open standards to get a foothold (just as x86 replace proprietary CPU architectures many decades ago), while still being acceptably capable. We shouldn't be throwing out devices because the manufacturer decided not to provide security patches for their OS variant, and certainly not because the manufacturer took away a feature we like through forced updates.</p> Google Nexus / Pixel (meh)</h3> I have run a google nexus</a> 4 previously which was flashable and rootable, but long since obsolete in hardware capabilities (and suffered a hardware failure). I never looked back at the google line after I discovered OnePlus, I can't comment on whether the current pixel phones are a good device for my constraints. I don't particularly want to further feed the beast. If you want the authentic pure google experience I guess these are probably worth a look but the pure google experience seems less and less appealing as the years roll on.</p> Sony (yuck)</h3> Someone told me that a particular Sony phone was flashable, I researched it, found instructions on flashing it, ordered one... and couldn't flash it. Discussing with Sony support it transpired that that precise model was sometimes flashable and sometimes not, and the only way to find out was to order one and look at the serial number. Fuck you Sony. They never have understood the open source movement and look like they still don't. They always treated their kit like appliances not a blend of hardware and software (or firmware) with everything that means for the long term management of the software. It's a shame because they do make nice hardware. I'd still buy a dumb Sony audio amplifier but not much else of theirs. Returned and refunded, try again...</p> OnePlus (yay, no wait, boo)</h3> This is my choice for the new OnePlus 9 Pro</a> I'm trying to set up, and the OnePlus 5T</a> I've been using for more than a couple of years now.</p> They don't make the grade for the pure "everything must be open including chip bytecode and chip architecture" as they use proprietary chips, but they make the right balance for me of reasonably high spec hardware for a reasonable price, android compatibility, and the ability to flash whatever ROM/OS you like (at your own risk).</p> I've been running a OnePlus 5t for a while now (bought outright, no network lockin or contract nonsense) and generally happy enough with it. There are some bugs and storage is starting to constrain hence the upgrade, plus the camera could be better by modern standards and I don't fancy carrying a separate camera.</p> I now have had the OnePlus 9 Pro sat in a drawer for two months waiting for me to figure out what-the-FSM</a> to run on it, hence this blog post. It's already been unlocked and I've tried flashing a couple of things but am not happy with the setup yet. I stopped looking at it while I concentrated all my efforts on making sure my business was going in the right direction (or any direction at all).</p> To get root you have to wipe the phone</p> </blockquote> The OnePlus comes with OxygenOS (an android derivative) preinstalled, but not rooted and with a locked bootloader. To get root you have to wipe the phone which means it's important to do it before you start using it. It's caused me no end of pain that you can't use titanium backup</a> to backup all your data without root, and you can't get root without losing your files. I understand why (so you can't just unlock a phone to steal all the data) but that doesn't make it less of a pain in the arse. Bonus points for operating system updates (over the air / OTA or just normal updates) de-root the phone.</p> XDA Developers detailed page on the OnePlus 9 including pro</a></p> But wait, there's trouble in paradise.</p> The founder Carl Pei left OnePlus in 2020</a> after 7 years at the company.</p> OnePlus was funded by an existing phone company called Oppo</a>. (And both OnePlus and Oppo are part of BBK</a>)</p> It seems that OnePlus is basically now dead in the water</a> as a brand if not a company in its own right and will be sucked back into whatever the mothership has to offer, breaking everything along the way by trying to merge Oppo's colorOS with OnePlus's OxygenOS into an unholy mess.</p> Fuck.</p> As if that wasn't enough:</p> "OnePlus is adding artificial limitations and breaking features via software updates, and there are no indications that they'll improve." ~ GCam Hub, https://www.celsoazevedo.com/files/android/google-camera/f/post-05/</a></p> </blockquote> It turns out that the founder of OnePlus, Carl Pei, has set up a new company called "Nothing</a>" (enjoy googling that one), which did just make headphones (unexciting), but who have just announced a phone. So maybe this is the true successor to the likely dead OnePlus brand. https://www.theverge.com/2022/3/23/22992424/nothing-phone-1-smartphone-carl-pei-apple-ecosystem</a></p> So what OS options does that leave?</h2> So not iOS and preferably not Android either.</p> Unfortunately the apps mean that I'm stuck with Android and its derivatives for the time being.</p> It seems the best hope for salvation is the Linux based options discussed above eventually getting sufficient app coverage to make the jump more bearable, or maybe even some kind of android emulation allowing them to run android apps sufficiently well. The path from Windows to Linux for me included a mix of dual-boot, virtualization and wine (an api compatibility layer, not an emulator)</p> It's worth mentioning that some (most/all?) phone manufacturers make proprietary customizations to Android or even complete forks</a> that you may or may not want. Samsung for example (for the Galaxy line of phones) replaces the "launcher" with the "Samsung Experience"</a>. I'm going for more open rather than less, so I'm not interested in trading slickness for losing even more control. I just want a platform that runs the apps I care about and is as open as possible.</p> Researching the options</h3> I don't follow this stuff all the time so I had to do some googling (duckduckgo</a>-ing). Here's some useful comparisons I turned up:</p> It's FOSS, Open Source Mobile OS Alternatives To Android</a> - mostly pure Linux things that don't have the android app ecosystem</li> PC Magazine, Break Away From Android: 7 Free Open-Source Mobile OSes to Try</a></li> GetDroidTips has some more ROMs I haven't looked at</a></li> </ul> I've got more to learn/research here so I'll expand this section as and when I learn more. Think of this as a bliki</a>. There's a full history in git (link at end).</p> Towering stacks of patches</h3> The below are probably the most realistic alternatives to stock Android and iOS for those who want more openness but don't want to give up all the modern conveniences in the name of openness and/or privacy. They are however all customizations of Android and that brings a very real problem.</p> The core open "Android Open Source Project (AOSP)" is run by google for google's benefit and for their own Android ecosystem. There is significant engineering effort continually poured in to this, with major releases made on remarkably short timelines.</p> Any project that makes a customization to AOSP such as LineageOS and keeps track of it in their own fork is constantly at the mercy of changes to the foundation it has been based on, a foundation which really has no reason to care about them.</p> This creates a never ending challenge that stunts innovation outside the Android/iOS duopoly. If you build a significant customization, it will sooner or later be broken and need re-engineering because of changes in AOSP, or maybe even be impossible to resurrect. The more you innovate and customize, the more engineering fire-power is required just to stand still on the treadmill of change in the platform.</p> While we have good tools for managing all this complexity (git, gerrit) this is a fundamental and unavoidable problem with building on a platform that doesn't care about you.</p> Contrast that with the Linux ecosystem where the platforms people are building on (consisting of many many layers and teams) are much more interested in supporting downstream projects and allowing choice. Projects that run on Linux can often run with minimal modification for decades. Linux is also not built for the benefit of some particular vendor like google with their own agenda, it is instead run by a foundation with far more broad interests and pressures.</p> Android derivative options</h3> I now need to look into the below and decide what to run (maybe with some trial flashing). Once that's done it'll be on to the slog of setting up the actual device how I want it.</p> OxygenOS from OnePlus - used to be good, now dying a painful death</h4> See the section on OnePlus hardware above. The company and OS are in trouble now the founder has left and the operating system has been merged with Oppo's colorOS (read "fucked")</p> https://www.oneplus.com/uk/oxygenos</a></li> </ul> This is the default for the OnePlus device I have, which means it's the most likely to make the hardware perform at its best.</p> After a wobble on data reporting</a> (which I'm not too bothered about) they seem to have largely sorted out their privacy game... apart from being full of google services just like the rest of the mainstream android devices. To play or not to play...</p> Here's the original article disclosing OnePlus silently sending data home</a>.</p> It seems the latest news is that Oxygen is falling apart. XDA Developers: "OxygenOS 12 for the OnePlus 9 series is littered with bugs"</a></p> "It was shit after merge with #Oppo and getting worst day by day. I really like #OnePlus device but #OxygenOS is dead now." ~ @rahulawanjari, 9 Dec 2021, twitter</a></p> </blockquote> Crap, there goes another good option. As if the mobile phone space wasn't horrific enough already.</p> In fact OxygenOS 12 was so bad, that they've just OnePlus pulled the plug on the entire OxygenOS 12 upgrade!!</a></p> Bonus points for the anti-competitive practice of actively denying access to hardware features (secondary cameras)</a>. I hope the courts slap them down for this clearly anti-competitive behaviour.</p> So after discovering that shit-show-in-paradise from OnePlus I could potentially skip OxygenOS 12 and stick to the less fcked OxygenOS 11, but that doesn't seem like much of a plan so it looks like I'll be skipping any fancy hardware capability and looking for a more open and less broken Android variant. Yet more proof that corporations can't be trusted as guardians of our software for the long term without GPL or at least MIT to stop them from hurting their users sooner or later. Did I mention my podcast is called "Software Should Be Free". Does this horror show give you a hint as to why it's called that?</p> LineageOS (yep)</h4> LineageOS</a> is the successor to the now defunct CyanogenMod. See the history of LineageOS and it's relationship to CyanogenMod here on Wikipedia</a>.</p> CyanogenMod was a bastian of freedom and innovation. I hope Lineage has managed to continue that.</p> There does appear to be a build of LineageOS for the OnePlus 9 pro</a> - I think this is worth trying out.</p> LineageOS has it's own recovery image, so no need to run TWRP as well.</p> Review of LineageOS on XDA Developers</a></p> Lineage for microG (maybe?)</h4> https://lineage.microg.org/</a></li> https://github.com/lineageos4microg</a></li> </ul> This is a fork of lineage with microG already set up properly and a patch that lineage wouldn't allow that let's them spoof google signatures in order to trick apps into believing they are using the real google play services</p> Q: "Why do we need a custom build of LineageOS to have microG? Can't I install microG on the official LineageOS?"</p> A: "MicroG requires a patch called "signature spoofing", which allows the microG's apps to spoof themselves as Google Apps. LineageOS' developers refused (multiple times) to include the patch, forcing us to fork their project." ~ Lineage for microG faq</a></p> </blockquote> Replicant (not yet)</h4> "Replicant</a> is a fully free Android distribution running on several devices" I'm glad this freedom is here but I'm not quite ready to give up every last proprietary thing until there's a bit less of a gap.</p> Supported devices</a> shows purely Samsung Galaxy devices currently.</p> /e/ (aka Eelo)</h4> https://e.foundation/</a></p> "We build desirable, open source, privacy-enabled smartphone operating systems."</p> </blockquote> This Android derivative is focussed on privacy; eliminating as much data collection (i.e. all things google) as possible. Laudable but not my main focus right now.</p> You can buy their reconditioned phones with their OS pre-installed or just install the OS on something else.</p> They provide their own alternative cloud hosting for contacts etc for reasonable prices.</p> Their dev build has a root option it seems which might be useful. - https://doc.e.foundation/support-topics/root-e-os.html</a></p> /e/ on wikipedia</a></li> itsfoss.com/eelo-mobile-os</a></li> https://www.pcmag.com/news/google-free-android-smartphones-are-now-available-in-the-us</a></li> </ul> GrapheneOS (nope)</h4> "GrapheneOS</a> - The private and secure mobile operating system with Android app compatibility. Developed as a non-profit open source project."</p> </blockquote> GrapheneOS only officially supports Pixel phones</a></p> CalyxOS (nope)</h4> CalyxOX</a> - "Your Phone Should Be Private Everyone needs a phone. Not everyone wants to be spied on. Reclaim your privacy with CalyxOS."</p> </blockquote> Supports Pixel devices and one Xiaomi phone (never heard of it).</p> Seems to be pushing Signal and Tor so is super-privacy focussed.</p> Google Apps and Services</h2> There are two major parts to this:</p> Google play - the app store for downloading apk files to install (plus ratings, reviews, screenshots etc)</li> Google services - these provide services that many apps want to use to avoid multiple implementations such as location information, push notification etc.</li> </ol> There's a few ways of getting the google play store and related proprietary horrors:</p> Open GApps - pulls down the proprietary apps and writes them to the image</li> MindTheGapps? - pulls down the proprietary apps and writes them to the image</li> microG - open source re-implementation of the services</li> </ul> For Lineage 18.1 (Android 11) the Lineage wiki links to MindTheGaps</a>, and XDA Developers says "Always Use the GApps Package Recommended by your ROM Developer"</a></p> I don't mind proprietary software as long as there's choice out there, and the apple app store + google play duopoly on apps is not choice. F-Droid is fine but doesn't have a single big name vendor's apps, so it's probably enough for google to dodge an anti-competitive lawsuit but not enough to produce any real competition.</p> Comparisons:</p> https://www.reddit.com/r/LineageOS/comments/8358p0/mindthegapps_vs_opengapps/</a></li> https://www.reddit.com/r/degoogle/comments/olsv4c/i_am_very_confused_by_opengapps_mindthegapps/</a></li> </ul> Open GApps (installer)</h3> Installer for google's proprietary services.</p> GApps https://wiki.lineageos.org/gapps</a></li> https://github.com/opengapps/opengapps</a></li> https://opengapps.org/</a></li> </ul> Q: Why is this such a pain? A: Licensing...</p> "Due to licensing restrictions, these apps do not come pre-installed with ROMs others than those from vendors that are part of the Open Handset Alliance</a> and must be installed as a sideload package by the user themselves." ~ https://github.com/opengapps/opengapps/wiki/FAQ</a></p> </blockquote> We're lucky they let us do this at all, google could take their toys away from us at any time.</p> For Open GApps pico looks sufficient https://github.com/opengapps/opengapps/wiki/Package-Comparison</a>, Lineage recommends nano and no bigger.</p> MindTheGapps (installer)</h3> Installer for google's proprietary services.</p> Really haven't found much information about this.</p> About</h4> https://gitlab.com/MindTheGapps/vendor_gapps</a> - I think this is probably the official source There's no download link, as mentioned in this issue: https://gitlab.com/MindTheGapps/vendor_gapps/-/issues/1</a></li> https://github.com/MindTheGapps/vendor_gapps</a> appears to be a stale copy, maybe they moved hosting to gitlab</li> </ul> </li> https://www.getdroidtips.com/mindthegapps-8-1-0/</a></li> https://forum.xda-developers.com/t/change-opengapps-for-mindthegapps-afterthought.3837816/</a></li> </ul> Installation</h4> This wasn't a lot of help but basically said just flash it: https://www.getdroidtips.com/mindthegapps-8-1-0/</a></li> </ul> microG (open source reimplementation)</h3> https://github.com/microg</a></p> A free software implementation of shared services provided by the proprietary google services (such as location and push messaging used by many apps).</p> Doesn't work properly when flashed to lineage according to the faq so they made their own fork of Lineage "Lineage for microG" (see above)</p> Aurora - play store proxy</h3> Download apks (installation files) from the google play store without any googleness.</p> https://aurora-store.en.uptodown.com/android</a></li> https://f-droid.org/packages/com.aurora.store/</a></li> </ul> Yalp - play store proxy</h3> https://yalp-store.en.uptodown.com/android</a></p> F-Droid app store</h3> F-Droid</a> is an app store for android like google play but without all the googlyness.</p> This a bit pointless if you install play because all the open source apps are available in both, and the closed ones are only in play. I think I'll skip this for now but I'm glad it exists... Though it is pre-installed in lineage-for-microG so I have poked around and installed a few things.</p> Playmaker - play store / f-droid integration</h3> https://github.com/NoMore201/playmaker</a></p> Recovery images</h2> TWRP (great but not needed)</h3> Previously I've used TWRP, and I got as far as flashing it before discovering Lineage has it's own recovery image</a></p> Unlock bootloader https://www.getdroidtips.com/oneplus-9-pro-unlock-bootloader/</a></li> Download TWRP for lemonadep https://twrp.me/oneplus/oneplus9pro.html</a> - 3.6 twrp download for lemonadep</a></li> Get twrp public gpg/pgp public key to be able to verify .asc</code> files https://twrp.me/faq/pgpkeys.html</a></li> Flash TWRP (TeamWin Recovery Project?) https://twrp.me/</a> Power off the phone</li> Hold down volume-down and power buttons</li> Select english</li> select reboot to fastboot</li> connect the usb cable (usb-c to usb-c doesn't work on my dell xps, use the bigger usb 3 A to usb-c cable)</li> run fastboot devices</code> on the laptop (connected over usb), you should see your phone listed</li> boot the phone from the local twrp image fastboot boot twrp-3.6.0_11-0-lemonadep.img</code>, phone boots up into twrp gui</li> advanced > flash current twrp</li> there's a note about bootloops, but the options isn't there so skipping that</li> reboot > power off (says current slot: A, just for the record)</li> power on with volume-down held again, boots straight to twrp this time</li> </ul> </li> </ul> LineageOS's recovery image</h3> Came across this here: https://www.xda-developers.com/lineageos-18-1-review/</a></p> Works well, flashed no problem and have used both recovery and fastboot modes with no issues. You'll see it as part of the steps below.</p> Installing LineageOS on OnePlus 9 Pro (take 1)</h2> It begins. LineageOS</a>.</p> The oneplus 9 pro device is codename lemonadep</code>.</p> Useful howtos:</p> LineageOS official installation wiki page for OnePlus 9 pro lemonadep</a></li> How-To Geek: How to Install LineageOS on Android</a></li> Android Authority: Beginner’s guide to installing LineageOS on your Android device</a> - lots more context and screenshots</li> LineageOSROMS (unofficial)</a> - a very short 8 step guide LineageOSROMS (unofficial) full guide</a></li> </ul> </li> GetDroidTips</a> - not sure there's much extra useful here</li> </ul> XDADevelopers coverage of Lineage support for OnePlus 9 pro</a></p> The steps:</p> Download latest nightly (there's no stable/unstable on this): https://download.lineageos.org/lemonadep</a> - this has both the ROM (OS image) and the recovery image.</li> Download copy-partitions https://androidfilehost.com/?fid=2188818919693768129</a> (as per wiki) The sha256 for the copy I have is 200877dfd0869a0e628955b807705765a91e34dff3bfeca9f828e916346aa85f copy-partitions-20210323_1922.zip</code></li> </ul> </li> Verify all the sha256 sums: sha256sum -c .sha256</code></li> Boot to fastboot</li> flash the lineage recovery:tim@max:~/Downloads/oneplus9pro/LineageOS</span></span> $ fastboot flash boot lineage-18.1-20211214-recovery-lemonadep.img</span></span> Sending 'boot_a' (196608 KB) OKAY [ 6.113s]</span></span> Writing 'boot_a' OKAY [ 0.655s]</span></span> Finished. Total time: 6.985s</span></span></code></pre></li> copy partitions as instructed boot into recovery (vol-down + power)</li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload copy-partitions-20210323_1922.zip</code></li> Ignore unknown sig and "continue"</li> </ul> </li> wipe boot into recovery (vol-down + power) if not already in it</li> "format data / factory reset"</li> </ul> </li> OS install boot into recovery (vol-down + power) if not already in it</li> "apply update" > "apply from adb" (aka sideload)tim@max:~/Downloads/oneplus9pro/LineageOS</span></span> $ adb sideload lineage-18.1-20211214-nightly-lemonadep-signed.zip</span></span> serving: 'lineage-18.1-20211214-nightly-lemonadep-signed.zip' (~47%)</span></span> adb: failed to read command: Success</span></span></code></pre>apparently this is normal, but the failed to flash error on the phone is not, and rebooting drops me back into the existing Oxygen install. Dammit.... Try again</li> Second attempt ran fine output: step 1/2</code></li> output: step 2/2</code></li> then recovery logo comes back up and it just sits there</li> volume up to the back arrow at the top, power to press it</li> then</em> it says "install complete with status 0</code>" (zero being unix-speak for no-issues)</li> but it doesn't ever get past the lineage logo when booting, maybe because of the boot of oxygen. hmm</li> hard-power-off</a> by holding down volume-up and power button</li> </ul> </li> take 3, back to recovery</li> re-wipe (factory reset)</li> reboot</li> this time it booted... briefly</li> it got to setup screen, then before I did anything rebooted into some kind of recovery failsafe and prompted to factory reset again, which I did. This time it rebooted and stayed up while I zipped through the wizard.</li> </ul> </li> </ul> </li> go take a look - ignore the warning about doing gapps etc first just so we can see what vanilla Lineage looks like. We can always wipe & reflash wonderfully empty app list!</li> basic main camera & selfie cam works</li> flashlight works</li> odd android verify notification, didn't seem to work</li> top shelf only available on lockscreen, odd</li> no sim yet and wifi not connected yet, so not much else to test</li> vol-down + power still takes screenshots</li> </ul> </li> </ul> Connect adb, see if we get root shell</h3> https://developer.android.com/studio/command-line/adb#Enabling</a></p> On the phone enable adb settings > about phone > tap "build number" repeatedly</li> settings > system > advanced (grrrr) > developer options usb debugging [on]</li> rooted debugging [on]</li> disable adb authorization timeout [on]</li> </ul> </li> connect usb cable to laptop</li> phone prompts "Allow USB Debugging?" tick "always allow"</li> press "allow"</li> </ul> </li> </ul> </li> On the laptop adb devices</code> should now show the phone</li> </ul> Success with non-root:</p> $ adb shell</span></span> OnePlus9Pro:/ $ ^D</span></span></code></pre> Root denied without flipping the "rooted" option:</p> $ adb root </span></span> ADB Root access is disabled by system setting - enable in Settings -> System -> Developer options</span></span></code></pre> After enabling rooted debugging as above:</p> $ adb root </span></span> restarting adbd as root</span></span> $ adb shell</span></span> OnePlus9Pro:/ # whoami</span></span> root</span></span></code></pre>Getting root back</h3> Grumble</h4> It really annoys me that on a laptop this isn't even a step, yet on Android this is a whole additional drama. But I may have already mentioned that. Understanding the reasons doesn't mean I have to like it.</p> adb root shell</h4> According to https://www.xda-developers.com/lineageos-dropping-superuser-addonsu-implementation-favor-magisk-manager/</a> adb root shell is the official LineageOS way of getting root access to "mess with important files".</p> $ adb root </span></span> restarting adbd as root</span></span> $ adb shell</span></span> OnePlus9Pro:/ # whoami</span></span> root</span></span></code></pre> Worth knowing but not much help for running titanium backup as root.</p> Magisk</h4> According to https://www.xda-developers.com/lineageos-dropping-superuser-addonsu-implementation-favor-magisk-manager/</a> Magisk is the only real option now. I guess SuperSU still works but lacks the masking that allows you to fool android pay etc.</p> ⚠️ Beware of fake Magisk sites !! ⚠️</h5> Github is the official home for Magisk and it's downloads: https://github.com/topjohnwu/Magisk</a></p> Beware of clone sites</strong></a> especially ones offering downloads. MagiskManager.com</code> is not legit or authorised and may host malware at some point. I've seen a few others too. They look like SEO optimised clones that could potentially be feeding some or all visitors malware.</p> It's worth being extra-careful with things like recovery images and rooters because by definition they have full control of your device, malicious or malign.</p> Magisk root install</h5> LineageOS said not to boot before doing installing GApps, I'm not sure if that applies to Magisk too. I already have booted up. Not sure if a factory reset plus wipe will be enough or whether the image will have to be reflashed. I'll try with just an install, if that doesn't work I'll try the reset and see what happens, and if that</em> doesn't work I'll try a full reinstall.</p> Look at readme at https://github.com/topjohnwu/Magisk</a></li> Follow instructions https://topjohnwu.github.io/Magisk/install.html</a></li> Download latest release https://github.com/topjohnwu/Magisk/releases/latest</a></li> install the app with adb (with the phone on, and booted to LineageOS as per normal use) https://www.droidviews.com/install-apk-files-using-adb-commands/</a></li> </ul> $ adb install Magisk-v23.0.apk </span></span> Performing Streamed Install</span></span> Success</span></span></code></pre> On the phone, swipe up for the full app list (drawer), and there is a Magisk icon!</li> </ul> </li> </ul> Getting boot.img</h6> The instructions for Magisk just say you need boot.img</code>, leaving know clues how to get it. I've proposed explaining how to extract them in this pull request</a></p> It turns out it's embedded in the Lineage system image lineage-18.1-20211214-nightly-lemonadep-signed.zip</code> that we've already downloaded.</p> There's another payload extractor with easier dependencies that's worth a try, especially if you don't already have python: payload-dumper-go</a> \| payload-dumper-go downloads</a>. I've now tested this and it works great. You don't even have to unzip the zip.</p> Extract the payload.bin</code> file from the lineage-18.1-20211214-nightly-lemonadep-signed.zip</code> to the same folder as the python script.</p> Some instructions on using python to extract the boot image: https://forum.xda-developers.com/t/guide-how-to-extract-payload-bin-from-ota.3830962/</a> (ignore the file it's older than the github one)</p> The payload_dumper</code> file appears to be from https://github.com/vm03/payload_dumper</a> assuming my googling is accurate.</p> I didn't need to do the pip install.</p> I did change the python to python3 at the top of the py file and make it executable.</p> tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span> $ ./payload_dumper.py payload.bin </span></span> boot</span></span> dtbo</span></span> odm</span></span> product</span></span> system</span></span> system_ext</span></span> vbmeta</span></span> vbmeta_system</span></span> vendor</span></span> vendor_boot</span></span> tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span> $ ll output </span></span> total 5.6G</span></span> -rw-rw-r-- 1 tim tim 192M Dec 20 23:20 boot.img</span></span> -rw-rw-r-- 1 tim tim 24M Dec 20 23:20 dtbo.img</span></span> -rw-rw-r-- 1 tim tim 2.8M Dec 20 23:20 odm.img</span></span> -rw-rw-r-- 1 tim tim 1.3G Dec 20 23:21 product.img</span></span> -rw-rw-r-- 1 tim tim 267M Dec 20 23:21 system_ext.img</span></span> -rw-rw-r-- 1 tim tim 2.5G Dec 20 23:21 system.img</span></span> -rw-rw-r-- 1 tim tim 8.0K Dec 20 23:21 vbmeta.img</span></span> -rw-rw-r-- 1 tim tim 4.0K Dec 20 23:21 vbmeta_system.img</span></span> -rw-rw-r-- 1 tim tim 192M Dec 20 23:22 vendor_boot.img</span></span> -rw-rw-r-- 1 tim tim 1.3G Dec 20 23:22 vendor.img</span></span></code></pre> Hurrah, now we have a boot.img</code> to give to Magisk.</p> Magisk rooting</h6> Copy the boot.img</code> across:</p> tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span> $ adb push output/boot.img /sdcard/ </span></span> output/boot.img: 1 file pushed, 0 skipped. 177.5 MB/s (201326592 bytes in 1.082s)</span></span></code></pre> Press install on the Magisk app.</li> "Select and Patch a File"</li> hamburger in file browser > "OnePlus 9 Pro" (obviously, duh) > boot.img</code></li> "let's go ->"</li> Magisk shows steps then" output file is written to..." and a path to download folder</li> </ul> tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span> $ adb shell</span></span> OnePlus9Pro:/ # ls -lh /sdcard/Download/</span></span> total 96M</span></span> -rwx------ 1 u0_a163 u0_a163 192M 2021-12-14 13:21 magisk_patched-23000_BYMDt.img</span></span></code></pre> pull the file back to the laptop adb pull /sdcard/Download/magisk_patched-23000_BYMDt.img</code></li> Now flash the file to the phone: (the vol-up + power didn't work for me, just did a normal boot, vol-down + power is reccovery not fastboot)</li> Settings > system > advanced (ffs) > gestures (wtf?) > power menu > advanced restart [on]</li> power button (hold) > power > restart > fastboot</li> </ul> </li> </ul> tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span> $ fastboot flash boot magisk_patched-23000_BYMDt.img </span></span> Sending 'boot_b' (196608 KB) OKAY [ 6.035s]</span></span> Writing 'boot_b' OKAY [ 0.639s]</span></span> Finished. Total time: 6.853s</span></span></code></pre> power off</li> power on</li> open up the Magisk app - it shows installed version number now, hurrah</li> </ul> Using Magisk:</p> Click the cog (top right) to optionally turn on "MagiskHide"</li> Along the bottom are four icons Home</li> Shield - shows root requests</li> bug - shows logs</li> jigsaw piece - shows modules, activate, deactivate & install</li> </ul> </li> </ul> adb</h3> adb</a></p> I consider this an important backup access to the phone. If the screen fails then this allows you to pull any important files off over usb.</p> Steps for this are above.</p> Installing "Lineage for microG" (tried, rejected for now)</h2> https://lineage.microg.org/</a> has install and download links</li> Download image from https://download.lineage.microg.org/lemonadep/</a> check sha: sha256sum -c lineage-18.1-20211220-microG-lemonadep.zip.sha256sum</code></li> run update_verifier python script: download (or clone) python update_verifier</a> - https://github.com/lineageos4microg/update_verifier/archive/refs/heads/master.zip</a></li> setup python3 with asdf-vm</a> (I had python 3 but no pip, and I like to use adsf)</li> extract the zip, open terminal in extracted folder</li> asdf plugin add python</code></li> asdf install python 3.10.1</code></li> asdf local python 3.10.1</code></li> pip install -r requirements.txt</code></li> python update_verifier.py lineageos4microg_pubkey ../lineage-18.1-20211220-microG-lemonadep.zip</code></li> output: verified successfully</code></li> </ul> </li> </ul> </li> Go back to lineage install instructions https://wiki.lineageos.org/devices/lemonadep/install</a></li> Boot to fastboot</li> flash the lineage-microG recovery: fastboot flash boot lineage-18.1-20211220-microG-lemonadep-recovery.img</code></li> copy partitions as instructed boot into recovery (vol-down + power) or "enter recovery" from lineage fastboot</li> </ul> </li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload ../LineageOS/copy-partitions-20210323_1922.zip</code></li> Ignore unknown sig and "continue"</li> </ul> </li> wipe boot into recovery (vol-down + power) if not already in it</li> "format data / factory reset"</li> </ul> </li> OS install boot into recovery (vol-down + power) if not already in it</li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload lineage-18.1-20211220-microG-lemonadep.zip</code></li> </ul> </li> reboot</li> skip all the guided setup steps</li> open the microG app</li> run the self check (all green ticks for me, woo)</li> </ul> So now I have a clean LineageOS install with a FOSS re-implementation of google's proprietary shared services. Win.</p> Now re-run magisk sideload from above to get root again.</p> re-enable adb as before</li> re-enable advance power button menu</li> adb install Magisk-v23.0.apk</code></li> grab a payload-dumper-go binary release</a></li> payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go lineage-18.1-20211220-microG-lemonadep.zip</code></li> adb push extracted_20211221_231517/boot.img /sdcard/Download/</code></li> run the patch in the phone Magisk UI (under "install")</li> get the filename to pull: adb shell ls /sdcard/Download</code></li> adb pull /sdcard/Download/magisk_patched-23000_d5mu3.img</code></li> reboot to fastboot</li> fastboot flash boot magisk_patched-23000_d5mu3.img</code></li> "reboot system now" in lineage fastboot menu</li> </ul> Success, now I have root and</em> microG.</p> Next, see if Aurora does the job, and if not how do I get google play on top of microG, if that's even possible. Till next time...</p> Getting apps</h3> F-Droid</h4> F-Droid is preinstalled (stared blankly at app list for a while before spotting it!) so that was easy, this can install many</em> open source apps, and warns about "anti-features" which is a nice touch.</p> Used F-Droid to install andOTP as a test (alternative to google authenticator 2FA app). Worked well.</p> Apps from play store - Aurora</h4> Found Aurora in f-droid with a search, installed no problems, opened it up.</p> First thing it does is prompt for install type which is confusing and lacks enough context to know what it means. The four types are explained further here: https://auroraoss.com/faq/#how-does-aurora-store-install-apps</a> thought I'm still not really sure. I don't know what a "split apk" is yet, and I don't see if there's a difference between root and services as I could do either and they both say they do the same thing.</p> Session - recommended but not clear why, I think it's saying it's like "Aurora Services" but with more something-or-other</li> Native - doesn't support split apks, that sounds bad, see below for what they are (see below for split apk research)</li> Root - auto install when downloaded</li> Aurora Services - auto install when downloaded</li> </ul> How to choose? I guess I'll go with the recommendation.</p> Couple of theme questions. Meh.</p> Installer question: "Select a suitable installer"</p> External Storage Access [grant] - pops and allow dialog, clicked "allow"</li> External Storage Manager [grant] - required on R+ (aka android 11), which we are running, guess I'll need that, - opened permissions screen, toggled to allow</li> Installer Permission [grant] - opened permissions screen, toggled to allow</li> </ul> "Finish"</p> Log in using:</p> Google</li> Anonymous</li> </ul> It seems from what I've read in the FAQ and terms that using the real login is best because the anon accounts end up with random locales and app restrictions depending on where the login was generated.</p> [install keepassdroid and push my kdb across with adb to get the google login, eventually the kdb will be sync'd with syncthing]</p> ... log in with google ...</p> And we're in, app listing showing. Disconcerting to see disney+ etc in "free" app store on a "free" phone.</p> WhatsApp isn't available in f-droid so let's try that as an experiment... It's in the list... it installed... it opened... it worked! (And logged me out of the other phone, fucking whatsapp).</p> What's a split APK?</h5> APK splitting - https://developer.android.com/studio/build/configure-apk-splits</a></li> Android App Bundles - defers apk generatoin to play store https://developer.android.com/guide/app-bundle</a></li> Feature delivery - partial app additions and updates https://developer.android.com/guide/playcore/feature-delivery</a></li> Old unsupported app "SAI" split apk installer: https://forum.xda-developers.com/t/guide-split-apk-installation.3934631/</a></li> https://github.com/Aefyr/SAI</a></li> </ul> </li> https://www.reddit.com/r/androiddev/comments/cenuvs/what_is_it_with_publishing_split_apks_on_google/</a></li> https://www.reddit.com/r/androiddev/comments/ci4a7r/confused_about_split_apk/</a></li> </ul> The state of the system</h3> What's working and what isn't.</p> Google voice to text (dead)</h4> The keyboard has a little mic button which allows you to use google's services to enter text anywhere you like by typing. This is useful as I hate phone keyboards. I use it quite a lot. Unsurprisingly with the microG setup that's not working at all. The button is there but it does nothing.</p> Bluetooth (working)</h4> Headphones connected and played no problem.</p> The state of the apps</h3> What's working and what isn't.</p> Banking apps (mixed)</h4> Let's start with the big one. This is most likely to sink an open source effort as they try to protect themselves by joining in the lockdown.</p> Surprisingly two out of three of the banking apps I'd like to use actually worked on the rooted custom ROM with an unlocked bootloader. I had not turned on the magisk hide capability at this point. One banking app behaved as if nothing was different, one popped a warning but allowed the app to continue and one flat out refused to run. I'm not going to name the banks involved on the public internet as I don't want to encourage anyone to test the security of my banking.</p> One of the apps logged me out of my other phone. The other functioning app offered to transfer a "digital key" to the new device, and without that allowed me to use the first phone to generate a login code and use that to poke around. I didn't try transferring the key.</p> More worrying was the realisation that I'd been locked in to running a google/iOS locked-down device by one of the banks as they are an "app-first" bank. You can't even use the online banking without the phone app, and the phone app requires a locked and signed phone. It's only by going on this very journey of actually trying to set up a phone that I control that I'd even noticed this insidious creeping embrace of platform-lockin. So now to truly escape the platform, I have to change banks as well.</p> Google calendar (mostly working)</h4> I'm afraid I still use google's calendar, and there's a high cost of change as I've got my other half using a shared calendar. The calendar worked fine, although oddly the "widget" (view of calendar that can be placed on the phone's desktop when unlocked) was blank until I left the phone in the drawer with wifi off for two days. For who-knows-what reason when I got it out the drawer it has started working.</p> Oddly the events have shown up but reminders are missing. This is a bit of a problem as I use them a lot. It's not that they're local to my other phone because they show on the google calendar web interface too.</p> Amazon things (working)</h4> The amazon shopping app worked no problem.</p> To my surprise the amazon primve video app worked and played content as normal. I rather expected some kind of DRM smackdown to kick in on a rooted device.</p> Audible worked and played no problem.</p> Spotify (working)</h4> Again pleasingly working with no quibbling or warnings. This is more important to me at the moment than perhaps it should be.</p> Casting sound to chromecast audio not working.</p> Whatscrapp (working)</h4> Much as I hate WhatsApp and it being owned by Facebook/Meta .... thanks to network effect I kinda still need it. I've managed to get some of my contacts over to telegram/signal but not everyone.</p> WhatsApp worked but signed me out of the other phone (ffs). Bonus points for receiving an important message from someone I haven't heard from in about a year on the wrong phone. Ugh.</p> So on the plus side I've learned of another reason to dislike WhatsApp.</p> Hiding the unlock & root</h3> Magisk has in its settings a "hide" option. This had no noticeable effect on the "SafteyNet" test (still three of three tests fail for me) built in to Magisk. I didn't notice any change in app behaiour in my limited poking.</p> WiFi-based location (turn on)</h3> Understandably from a freedom-warrior point of view the wifi-based loction services are off by default, but this is a bit of a pain because if you've ever used pure GPS positioning you'll know it's shit at doing it in houses, offices and built up areas, takes ages to get a lock, and is a bit hit and miss in cars. The accuracy you normally see is much more a function of the ubiquity of wifi in our urban lives.</p> Screen refresh rate 60/120Hz (increase)</h3> The "minimum screen refresh rate" setting was set to 60Hz. There's also a developer option for showing the current rate. In my experiment it sat at 60Hz all the time. This is fine till you scroll up and down, then you notice how much smoother the text scrolls at 120Hz. I upped the minimum to 120Hz which seemd to work.</p> There's also a smooth-scroll option that I turned on that seems to produce a nicer effect.</p> Using adb to do backups</h2> It might be possible to backup a locked down phone without root by using adb's backup capabilities. I haven't looked into this much or tried it yet but here's some quick research notes:</p> gist: Backup android app, data included, no root needed, with adb</a></li> https://net2.com/how-to-backup-and-restore-your-android-device-with-adb-on-ubuntu/</a></li> https://www.paulligocki.com/backup-restore-android-with-adb/</a></li> https://stackpointer.io/mobile/android-adb-backup-extract-restore-repack/372/</a></li> https://stackoverflow.com/questions/19225467/backing-up-android-device-using-adb</a></li> https://android.stackexchange.com/questions/69567/what-all-does-adb-backup-and-how-do-i-restore-part-of-it</a></li> </ul> This might give me a way of rescuing some of my data from my old phone that de-rooted itself in an OS upgrade and fucked my titanium backups (still bitter).</p> Oh but wait...</p> Devs can set the ALLOW_BACKUP flag to "No"</a></p> </blockquote> Argh, what is wrong with this platform.</p> Customisations</h2> Turn on all the power button options: Settings > system > advanced (ffs) > gestures (wtf?) > power menu > advanced restart [on]</li> </ul> </li> </ul> Options - what now?</h2> That's been an enlightening exercise in "is this even possible". I've learned a huge amount about the current state of the phone ecosystem. I'm in equal measure dismayed and hopeful, there are some unpleasant trends and circumstances, and some really encouraging signs of hope for the open ecosystem.</p> There really isn't a clear answer for me for where to go from here. There are downsides and challenges in every direction.</p> The options available seem to be in increasing order of freedom and somewhat decreasing order of utility:</p> Get an iPhone and abandon my principles and control, go live in a digital nursing home. - YeahNo. get banking apps galore</li> never get root again</li> trust apple to administer the correct dosage of backups</li> forever more fight the divide between iOS and my linux laptops</li> </ul> </li> Install Oxygen 11, relock the bootloader, live without root, have no upgrade path because Oxygen is dead. - No. possibly brick the phone trying to lock it</li> get banking apps</li> never get root again</li> never have decent backups</li> forever more fight the divide between iOS and my linux laptops</li> </ul> </li> Install lineage and one of the google apps installers still no bank</li> probably get reliable google goodness</li> get stalked by google (no change there, I dno't mind as much as some about this)</li> unknown time-to-failure</li> I think this is probably the most "mainstream" of the subversive options so is most likely to continue to have a forward path for the next few years</li> worry about security of unlocked evenrything and the security of lineage - I think it's probably no worse than linux at this point but who knows</li> </ul> </li> Install lineage-for-microG (current experiment) no bank, no voice-to-text</li> worry more about the security of build servers for this niche option</li> I don't know how sustainable this project is, it's certainly noble</li> </ul> </li> go all in and run a linux distro definitely no apps for you</li> get to take the moral high ground</li> lose touch with everyone who is on whatsapp</li> </ul> </li> </ul> It would be nice to think that by choosing to run the most open thing I can I am in some way helping, but I think the truth is users really don't make much difference to projects like this unless they come in enough numbers to sway app vendors like WhatsCrapp to support more obscure platforms. It's only the hackers and those who can financially fund projects who really make a difference here. So while I want to run the most open thing I can it doesn't really matter a hill of beans outside of my own brain whether I do or not.</p> I think having written down the options I'm swinging towards lineage + proprietary google services. This seems to solve the immediate problem of getting unlocked and rooted and getting working backups again without losing everything that's useful to me in the process.</p> I'm glad I've tried the microG version, and hope we see open source slowly chip away at the power of the two giants just like Linux did in the desktop space over decades.</p> Installing LineageOS + google services on OnePlus 9 Pro (take 2) - fail</h2> I now have the lineage bootloader so the steps to get into recovery etc are a bit different than with stock or TWRP.</p> "Note: If you want the Google Apps add-on on your device, you must follow this step before booting into LineageOS for the first time!" ~ https://wiki.lineageos.org/devices/lemonadep/install#installing-lineageos-from-recovery</a></p> </blockquote> Download latest nightly (there's no stable/unstable on this): https://download.lineageos.org/lemonadep</a> - this has both the ROM (OS image) and the recovery image.</li> Download copy-partitions https://androidfilehost.com/?fid=2188818919693768129</a> (as per wiki) The sha256 for the copy I have is 200877dfd0869a0e628955b807705765a91e34dff3bfeca9f828e916346aa85f copy-partitions-20210323_1922.zip</code></li> </ol> </li> Verify all the sha256 sums: sha256sum -c .sha256</code></li> Boot to fastboot</li> flash the lineage recovery: fastboot flash boot lineage-18.1-20211228-recovery-lemonadep.img</code> don't reboot yet</li> </ol> </li> copy partitions as instructed switch to recovery</li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload copy-partitions-20210323_1922.zip</code></li> Ignore unknown sig and "continue"</li> come back out of recovery menu (don't reboot yet)</li> don't reboot yet</li> </ol> </li> "factory reset" > "wipe all data" don't reboot yet</li> </ol> </li> flash OS: "apply update" > "apply from adb" (aka sideload)</li> adb sideload lineage-18.1-20211228-nightly-lemonadep-signed.zip</code></li> don't reboot yet</li> </ol> </li> MindTheGapps - I haven't found any instructions for this anywhere check the arch (it's arm64</code>) here https://wiki.lineageos.org/devices/lemonadep/</a></li> Download the arm64 build from https://androidfilehost.com/?w=files&flid=322935</a> or the "mirror" http://downloads.codefi.re/jdcteam/javelinanddart/gapps</a></li> </ol> </li> Given there's no shas anywhere I downloaded from several mirrors and cross-checked the hashes. 85481cb98c8a8692f52c033ead1db436870af385498a917701fcd8c6182e145c MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li> ignore signature warning (the price of escaping a closed ecosystem)</li> error shown on phone, roughly:low resource device detected, removing large extras</span></span> not enough space for gapps! aborting</span></span> ...</span></span> error in /sideload/package.zip (status 1)</span></span></code></pre> patch that adds error message</a></li> current message on sigma</code> branch</a></li> In lineage recovery, mount sytsem</code> and enable adb, run df:OnePlus9Pro:/ # df -h /mnt/system </span></span> Filesystem Size Used Avail Use% Mounted on</span></span> /dev/block/dm-2 0.9G 0.9G 3.0M 100% /mnt/system</span></span></code></pre>That is indeed quite full. Annoyingly the installer doesn't say which partition is full.</li> Some ideas here https://forum.xda-developers.com/t/q-insufficient-storage-space-in-system-partition.3018464/</a></li> Try format & reboot to recovery, nope.</li> hint that it could be slot related https://www.reddit.com/r/LineageOS/comments/fuykda/lineage_os_171_opengapps_error_not_sufficient/</a></li> let's try a swap slot as per https://www.reddit.com/r/LineageOS/comments/fqashj/difficulty_installing_gapps_after_lineage_os/fluhd8s/?context=3</a></li> enter fastboot</li> </ol> tim@max:~/Downloads/oneplus9pro/MindTheGapps</span></span> $ fastboot set_active other</span></span> Setting current slot to 'b' OKAY [ 0.043s]</span></span> Finished. Total time: 0.044s</span></span></code></pre> enter recovery</li> re-run the sideload</li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li> ignore signature warning (the price of escaping a closed ecosystem)</li> same space error. sigh</li> </ol> </li> </ol> </li> </ol> Possible cause of space error</h3> Partition not mounted at all</a></li> LineageOS image has partitions that are too small</a> to fit extra g-apps in. Maybe use TWRP BigSys</a> to get bigger partitions??</li> Manually re-partition? Alegedly dangerous. https://android.stackexchange.com/questions/216123/android-how-to-increase-system-partition-and-decrease-data-partition#comment276924_216123</a> warning of brickability if you touch wrong device</li> https://www.reddit.com/r/Android/comments/2o8lvf/why_cant_we_resize_partitions_on_android_but_on/</a></li> </ul> </li> </ul> </li> Incompatibility with Lineage bootloader - use TWRP instead</a> - worth a try I think, probably the next thing I'll try.</li> Ruled Out: Wrong/stale slot in use</a> - tried swap slot (above), no change.</li> Ruled out: Booting phone before gapps install (didn't do this so not this).</li> </ol> Patching the installer - abandoned</h3> It seems to me having read around that the error message is hiding useful information on the state of the device and what exactly failed. (Which partition is out of space? Is it just read only or something?). I'n not feeling like playing with partitioning given the warnings I saw on a forum of potentially bricking a device</p> There's links to the source above and I did start fiddling with this, but you have to figure out how to re-sign the zip to get it to install at which point I fell down an android-sized rabbit hole.</p> Signing zips and apks</h4> Here's what I found so far, I haven't got it working yet.</p> I think</em> it's the same signing that you have to do to apk</code> files, and apk</code> is just a special shaped zip</code> file. Off into learning-android-sdk-land I went...</p> There's some kind of strange android app that can sign zips called ZipSigner</a> ZipSigner is missing source code</a> so you are relying on random binaries. Dodgy but probably well intentioned.</li> </ul> </li> https://android.stackexchange.com/questions/222262/how-to-create-manually-create-my-own-%e1%b4%8f%e1%b4%9b%e1%b4%80-update-file-to-be-used-for-adb-sideload</a></li> https://android.stackexchange.com/questions/95425/update-zip-just-for-fixing-file-permissions-possible</a></li> https://forum.xda-developers.com/t/tutorial-the-updater-script-completely-explained.2377695/</a></li> https://www.bettermobileapp.com/article/10558021/Updater-script</a></li> https://android.stackexchange.com/questions/191043/edditing-updater-script</a></li> https://androidforums.com/threads/custom-rom-updater-scipt-troubleshooting.701752/</a></li> https://android.stackexchange.com/questions/35600/how-to-create-an-update-zip-file-that-can-copy-rename-a-file</a></li> https://developer.android.com/studio/command-line/zipalign</a></li> https://developer.android.com/studio/publish/app-signing#opt-out</a></li> https://developer.android.com/studio/command-line/apksigner</a></li> https://www.addictivetips.com/mobile/what-is-zipalign-in-android-and-how-it-works-complete-guide/</a> https://stackoverflow.com/questions/3994035/what-is-aligned-memory-allocation</a></li> </ul> </li> https://www.androidcentral.com/installing-android-sdk-windows-mac-and-linux-tutorial</a></li> </ul> Let's try TWRP+Lineage+MindTheGapps - fail</h3> Install TWRP</h4> https://www.getdroidtips.com/replace-lineageos-recovery-twrp/</a></p> Reboot phone to fastboot or something (the built-in one where the volume key changes options), leave it in that state</li> fastboot devices</code> on linux should show the phone</li> fastboot boot twrp.img</code> (already downloaded and symlinked above)</li> Phone reboots into twrp temporarily</li> Advanced > flash current twrp</li> Reboot (ok the no-OS warning)</li> Yay, booted into twrp, comforting</li> </ol> Install LineageOS (yet again)</h4> Well, let's get a new nightly</li> Instructions say check model... Reboot to twrp (recovery) with USB connected to laptop</li> Laptop pops window for new mount mtp://OnePlus_LE2123_dade278d/</code></li> Yep, LE2123 is in the supported list. Phew.</li> </ul> </li> Skip all the "recovery" steps in https://wiki.lineageos.org/devices/lemonadep/install</a></li> I'm guessing I don't need to run "copy partitions" again because that's to avoid only having one working slot, and I already have to having done this before. Tell me if I'm wrong!</li> Give up waiting for new image download (~1GB), continue with older image...</li> Factory reset TWRP > Wipe > Swipe for factory reset</li> back</li> Format Data (button)</li> type yes to continue (wiping all apps & data)</li> back to main menu (android back button lots of times)</li> </ul> </li> Flash lineage Still in TWRP</li> Advanced > ADB Sideload</li> Tick the wipe boxes ("Dalvik cache" + "cache")</li> Swipe to sart sideload</li> adb sideload lineage-18.1-20211228-nightly-lemonadep-signed.zip</code></li> back</li> Don't reboot yet</li> </ul> </li> Add MindTheGaps still in TWRP</li> ADB Sideload</li> swipe (cache wipe still selected)</li> adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li> fail. log:</li> </ul> </li> </ul> *******************</span></span> MindTheGapps installer</span></span> *******************</span></span> Extracting files</span></span> Setting up environment</span></span> Mounting partitions</span></span> Could not mount /mnt/system! Aborting</span></span> Updater process ended with ERROR: 1</span></span></code></pre> For goodness sake.</p> Fuck it, let's try the other gapps even though it says not to....</p> Reboot just to have a look... Lineage running fine, no sign of any gapps.</p> Let's try TWRP+Lineage+OpenGApps - fail</h3> Power off</li> power + vol-down - ended up in lineage recovery, wat?? swap slot??</li> reboot to bootloader? no that's the built-in pixelated bright green "START" thing, oh actually that's fine, that's the fastboot thing.</li> switch slot as above</li> </ul> fastboot set_active other</span></span> Setting current slot to 'b'</span></span> ... OKAY [ 0.048s]</span></span> Finished. Total time: 0.056s</span></span></code></pre> reboot to recovery</li> back in TWRP</li> </ul> That's cool! I have lineage recovery in slot a</code> and TWRP in slot b</code>.</p> Okay, back to the plan.</p> Lineage (again again again...)</h4> Repeat the lineage intall steps above (no idea if that's really necessary but shrug</em>)</li> </ul> OpenGApps install - fail</h4> Download https://opengapps.org/?api=11.0&variant=nano</a> - ARM64 / 11 / nano (manually changed from 10 to 11 from link on https://wiki.lineageos.org/gapps</a>)</li> download the md5 too</li> md5sum -c .md5</code> to validate...</li> </ul> </li> Back to ADB Sideload in TWRP (cache wipe ticked x2), swipe to start</li> adb sideload open_gapps-arm64-11.0-nano-20220215.zip</code></li> cross fingers</li> bah. failed, basically same error.</li> </ul> - Performing system space calculations</span></span> </span> Insufficient storage space available in</span></span> System partition. ...</span></span> ...</span></span> Error Code: 70</span></span></code></pre> grap logs as instructed: adb pull /sdcard/open_gapps_log.txt</code> - success</p> gist of log file open_gapps_log.txt</a></p> Important bit here is that there is ZERO</strong> space, so choosing smaller install is not going to work.</p> ...</span></span> Current Free Space \| 0</span></span> ...</span></span></code></pre> Well on the plus side the diagnostics are muuuuch better in OpenGApps so that's saved me fucking around trying to patch MindTheGaps' installer.</p> Two-phone-Timmy - the burner phone</h2> Having spent some time completely stumped by the space issue and giving it some thought I realize that this battle with Android will never end. Trying to get an unholy mix of Libre and proprietary to mix is like oil and water.</p> I can opt out of that battle by running two phones.</p> A "primary" phone that I use as much as possible, running the Free-est thing I can muster. This would get the sim card. This would be the OnePlus 9 pro, ideally running PureOS as per the Librem 5 phone, but if not that then Lineage without the GApps that I couldn't get to work.</li> A burner phonee that's basically stock Android so that I can fall back to this for everything that can't or won't work on a free-er & unlocked base OS. (My existing OnePlus 5t for now while it lasts, reset to stock OxygenOS). This one can the get internet by tethering to the primary. I don't think any of the apps actually require the sim to be present. I guess we'll see.</li> </ol> The primary (9 Pro)</h3> The idea here is to get the free-est possible base layer (i.e. operating system), and then get maximum functionality within that. Anything that doesn't work or I can't find a working alternative to gets punted to the burner phone while we wait for hell to freeze over again. This will be the compromise for now.</p> Research trying PureOS, see what the app ecosystem is like - done - nope research done, doesn't seem to be possible to install on the OnePlus. Next...</li> </ul> </li> Continue with Lineage (I left the phone with a working but empty install of this)</li> Don't install proprietary google play services or play store at all.</strong> - This is the line in the sand. I would have accepted it for now, but given I couldn't install it, I'll take it as the new battle line for free vs proprietary.</li> Install microG for re-implementation of shared services (i.e. location, polling for updates from servers etc)</li> Install Aurora play store proxy again to be able to try out app store apps</li> See how close I can get to a usable phone.</li> </ul> The burner (5t)</h3> Factory reset the 5t</li> Install just what I couldn't get to work on the primary</li> Cry because the backups were shit and now I've lost access to something I care about (I predict)</li> </ul> Primary setup - experimenting</h2> Fired up the lineage (minus all google stuff) that I'd installed and then failed to get any gapps on (space issue above, no longer care).</p> Went through language prompt, was looking at time prompt in setup wizard when the phone rebooted with some kind of couldn't read data error. Disturbing. Rebooted, selected "try again" and it didn't do it this time.</p> Setup wizard</h3> - selected UK location,</span></span> - GMT timezone.</span></span> - Put in wifi password (incorrectly)</span></span> - set up pin</span></span> - set up fingerprint</span></span> - booted into OS (yay, pleasing to see Lineage in action)</span></span></code></pre>OS update with on-phone updater</h3> settings > advanced > updater</p> shows list of OS updates. hit refresh, nothing new. latest is 18.1 26th Apr 2022</p> tap download ....</p> tap intall (even longer pause ... installing ... finalising package installation ...) ..</p> reboot (press button that's presented)</p> config</h3> turn off annoying sounds settings > sound > advanced</li> screen locking sound - off</li> touch sounds - off</li> </ul> </li> </ul> todo</h3> replacement contact management (currently using google)</li> replacement calendar management (currently using google)</li> insert sim</li> </ul> already google-free</h3> email (fastmail domain + k9mail app)</li> broswer - firefox</li> search - duckduckgo</li> OTP - andOTP</li> maps/satnav (partially) - could use waze + maps.me (google maps is a decent car satnav, not found a perfect equivalent)</li> </ul> burner phone</h2> No sim in this one, use wifi and/or bluetooth tethering to get internet on it.</p> Things I can't escape the googly jail for</p> google maps?</li> apps for clients microsoft outlook</li> securid (vpn stuff)</li> google authenticator (use andOTP instead wherever possible)</li> microsoft authenticator (yuck, what's wrong with the OTP standard?!?!)</li> digidentity</li> </ul> </li> banking apps that don't work on the primary phone (not naming because haxxors)</li> google pay (never used it, just use an actual card)</li> google home for controlling the chromecast</li> spotify just for casting to the chromecast audio</li> </ul> To my horror I have discovered that android allows apps to force remote admin privs so that IT departments can snoop on your BYOD. Hello Microsoft Outlook + Active Directory. Yet more erosion of user rights in the name of security. There's no way in hell I'm allowing a client to be an admin on my primary phone so this is burner phone territory if at all necessary; though I just won't install any of that junk at all and will be uncontactable within their network unless on their supplied devices. Shrug.</p> Primary phone - microG</h2> lineage+microG - failed install</h3> Re-run setup in section: Installing “Lineage for microG”</p> OS install power off</li> boot into lineage recovery (vol-down + power) (currently slot "b" is active)</li> "apply update" > "apply from adb" (aka sideload)</li> adb sideload lineage-18.1-20211220-microG-lemonadep.zip</code> it seems this is flashed to slot a and then slot is automatically swapped</li> </ul> </li> ignore signature warning</li> accept downgrade warning</li> completed</li> back back</li> factory reset</li> reboot</li> drops to bootloader thing</li> "start" (power button to accept)</li> loops back to bootloader thing. damn.</li> vol down vol down - Recovery - power button to accept</li> starts lineage recovery - "Active slot: a" - hmmm</li> </ul> </li> </ul> try swap slot</p> $ fastboot set_active other</span></span> Setting current slot to 'b' (bootloader) Changing the active slot with a snapshot applied may cancel the update.</span></span> OKAY [ 0.038s]</span></span> Finished. Total time: 0.040s</span></span></code></pre> power off, power on</p> that worked, now enters the lineage setup steps</p> oh but it's back to the blank OS, the apr 2022 version!.</p> guess there's a slot for two OSes!</p> swap slot again, back to the bootloader loop. interesting.</p> let's try installing again</p> fastboot set_active other</span></span> Setting current slot to 'a' OKAY [ 0.060s]</span></span> Finished. Total time: 0.061s</span></span></code></pre>tim@max:~/Downloads/oneplus9pro/lineage-for-microG </span></span> $ adb sideload lineage-18.1-20211220-microG-lemonadep.zip </span></span> serving: 'lineage-18.1-20211220-microG-lemonadep.zip' (~21%)</span></span></code></pre> factory reset</p> reboot</p> cross fingers. nope. still just drops back to bootloader</p> recovery > factory reset > reboot</p> nope still fooked reboot to bootloader which says "Active slot: b" which is a bit odd. Maybe it does a swap slot automatically after flashing?</p> so at this point slot a has a fooked lineage-microG (maybe) and slot b has a functioning but blank vanilla lineage. I think.</p> https://www.reddit.com/r/GooglePixel/comments/8cz6m9/google_pixel_bootslot_b_brick/</a></p> grab the vars with fastboot getvar all 2>&1 \| sort > fastboot-getvar.txt</code></p> output (sensitive things redacted with XXXX) - interesting but not super helpful:</p> all: </span></span> (bootloader) battery-voltage:0</span></span> (bootloader) cpu-abi:arm64-v8a</span></span> (bootloader) current-slot:b</span></span> (bootloader) dynamic-partition:true</span></span> (bootloader) first-api-level:30</span></span> (bootloader) has-slot:abl_log:no</span></span> (bootloader) has-slot:abl:yes</span></span> (bootloader) has-slot:ALIGN_TO_128K_1:no</span></span> (bootloader) has-slot:ALIGN_TO_128K_2:no</span></span> (bootloader) has-slot:android_log:no</span></span> (bootloader) has-slot:aop:yes</span></span> (bootloader) has-slot:apdp_full:no</span></span> (bootloader) has-slot:apdp:no</span></span> (bootloader) has-slot:bluetooth:yes</span></span> (bootloader) has-slot:boot:yes</span></span> (bootloader) has-slot:carrier:no</span></span> (bootloader) has-slot:cdt:no</span></span> (bootloader) has-slot:connsec:no</span></span> (bootloader) has-slot:cpucp:yes</span></span> (bootloader) has-slot:ddr:no</span></span> (bootloader) has-slot:devcfg:yes</span></span> (bootloader) has-slot:devinfo:no</span></span> (bootloader) has-slot:dinfo:no</span></span> (bootloader) has-slot:dip:no</span></span> (bootloader) has-slot:DRIVER:no</span></span> (bootloader) has-slot:dsp:yes</span></span> (bootloader) has-slot:dtbo:yes</span></span> (bootloader) has-slot:engineering_cdt:yes</span></span> (bootloader) has-slot:featenabler:yes</span></span> (bootloader) has-slot:frp:no</span></span> (bootloader) has-slot:fsc:no</span></span> (bootloader) has-slot:fsg:no</span></span> (bootloader) has-slot:hyp_log:no</span></span> (bootloader) has-slot:hyp:yes</span></span> (bootloader) has-slot:imagefv:yes</span></span> (bootloader) has-slot:kernel_log:no</span></span> (bootloader) has-slot:keymaster:yes</span></span> (bootloader) has-slot:keystore:no</span></span> (bootloader) has-slot:limits-cdsp:no</span></span> (bootloader) has-slot:limits:no</span></span> (bootloader) has-slot:logdump:no</span></span> (bootloader) has-slot:logfs:no</span></span> (bootloader) has-slot:mdcompress:no</span></span> (bootloader) has-slot:mdtpsecapp:yes</span></span> (bootloader) has-slot:mdtp:yes</span></span> (bootloader) has-slot:metadata:no</span></span> (bootloader) has-slot:misc:no</span></span> (bootloader) has-slot:modemdump:no</span></span> (bootloader) has-slot:modemst1:no</span></span> (bootloader) has-slot:modemst2:no</span></span> (bootloader) has-slot:modem:yes</span></span> (bootloader) has-slot:multiimgoem:yes</span></span> (bootloader) has-slot:ocdt:no</span></span> (bootloader) has-slot:odm_b-cow:no</span></span> (bootloader) has-slot:odm:yes</span></span> (bootloader) has-slot:oplusdycnvbk:no</span></span> (bootloader) has-slot:opluslog:no</span></span> (bootloader) has-slot:oplusreserve1:no</span></span> (bootloader) has-slot:oplusreserve2:no</span></span> (bootloader) has-slot:oplusreserve3:no</span></span> (bootloader) has-slot:oplusreserve4:no</span></span> (bootloader) has-slot:oplusreserve5:no</span></span> (bootloader) has-slot:oplus_sec:yes</span></span> (bootloader) has-slot:oplusstanvbk:yes</span></span> (bootloader) has-slot:param:no</span></span> (bootloader) has-slot:persist_bkp:no</span></span> (bootloader) has-slot:persist:no</span></span> (bootloader) has-slot:product_b-cow:no</span></span> (bootloader) has-slot:product:yes</span></span> (bootloader) has-slot:qmcs:no</span></span> (bootloader) has-slot:qsee_log:no</span></span> (bootloader) has-slot:qupfw:yes</span></span> (bootloader) has-slot:qweslicstore:yes</span></span> (bootloader) has-slot:rawdump:no</span></span> (bootloader) has-slot:rtice:no</span></span> (bootloader) has-slot:sda:no</span></span> (bootloader) has-slot:sdb:no</span></span> (bootloader) has-slot:sdc:no</span></span> (bootloader) has-slot:sdd:no</span></span> (bootloader) has-slot:sde:no</span></span> (bootloader) has-slot:sdf:no</span></span> (bootloader) has-slot:secdata:no</span></span> (bootloader) has-slot:shrm:yes</span></span> (bootloader) has-slot:splash:yes</span></span> (bootloader) has-slot:spunvm:no</span></span> (bootloader) has-slot:ssd:no</span></span> (bootloader) has-slot:storsec:no</span></span> (bootloader) has-slot:super:no</span></span> (bootloader) has-slot:system_b-cow:no</span></span> (bootloader) has-slot:system_ext_b-cow:no</span></span> (bootloader) has-slot:system_ext:yes</span></span> (bootloader) has-slot:system:yes</span></span> (bootloader) has-slot:tzsc:no</span></span> (bootloader) has-slot:tz:yes</span></span> (bootloader) has-slot:uefisecapp:yes</span></span> (bootloader) has-slot:uefivarstore:no</span></span> (bootloader) has-slot:userdata:no</span></span> (bootloader) has-slot:vbmeta_system:yes</span></span> (bootloader) has-slot:vbmeta_vendor:yes</span></span> (bootloader) has-slot:vbmeta:yes</span></span> (bootloader) has-slot:vendor_b-cow:no</span></span> (bootloader) has-slot:vendor_boot:yes</span></span> (bootloader) has-slot:vendor:yes</span></span> (bootloader) has-slot:vm-bootsys:yes</span></span> (bootloader) has-slot:vm-data:no</span></span> (bootloader) has-slot:xbl_config:yes</span></span> (bootloader) has-slot:xbl:yes</span></span> (bootloader) hw-revision:0</span></span> (bootloader) is-logical:abl_a:no</span></span> (bootloader) is-logical:abl_b:no</span></span> (bootloader) is-logical:abl_log:no</span></span> (bootloader) is-logical:ALIGN_TO_128K_1:no</span></span> (bootloader) is-logical:ALIGN_TO_128K_2:no</span></span> (bootloader) is-logical:android_log:no</span></span> (bootloader) is-logical:aop_a:no</span></span> (bootloader) is-logical:aop_b:no</span></span> (bootloader) is-logical:apdp_full:no</span></span> (bootloader) is-logical:apdp:no</span></span> (bootloader) is-logical:bluetooth_a:no</span></span> (bootloader) is-logical:bluetooth_b:no</span></span> (bootloader) is-logical:boot_a:no</span></span> (bootloader) is-logical:boot_b:no</span></span> (bootloader) is-logical:carrier:no</span></span> (bootloader) is-logical:cdt:no</span></span> (bootloader) is-logical:connsec:no</span></span> (bootloader) is-logical:cpucp_a:no</span></span> (bootloader) is-logical:cpucp_b:no</span></span> (bootloader) is-logical:ddr:no</span></span> (bootloader) is-logical:devcfg_a:no</span></span> (bootloader) is-logical:devcfg_b:no</span></span> (bootloader) is-logical:devinfo:no</span></span> (bootloader) is-logical:dinfo:no</span></span> (bootloader) is-logical:dip:no</span></span> (bootloader) is-logical:DRIVER:no</span></span> (bootloader) is-logical:dsp_a:no</span></span> (bootloader) is-logical:dsp_b:no</span></span> (bootloader) is-logical:dtbo_a:no</span></span> (bootloader) is-logical:dtbo_b:no</span></span> (bootloader) is-logical:engineering_cdt_a:no</span></span> (bootloader) is-logical:engineering_cdt_b:no</span></span> (bootloader) is-logical:featenabler_a:no</span></span> (bootloader) is-logical:featenabler_b:no</span></span> (bootloader) is-logical:frp:no</span></span> (bootloader) is-logical:fsc:no</span></span> (bootloader) is-logical:fsg:no</span></span> (bootloader) is-logical:hyp_a:no</span></span> (bootloader) is-logical:hyp_b:no</span></span> (bootloader) is-logical:hyp_log:no</span></span> (bootloader) is-logical:imagefv_a:no</span></span> (bootloader) is-logical:imagefv_b:no</span></span> (bootloader) is-logical:kernel_log:no</span></span> (bootloader) is-logical:keymaster_a:no</span></span> (bootloader) is-logical:keymaster_b:no</span></span> (bootloader) is-logical:keystore:no</span></span> (bootloader) is-logical:limits-cdsp:no</span></span> (bootloader) is-logical:limits:no</span></span> (bootloader) is-logical:logdump:no</span></span> (bootloader) is-logical:logfs:no</span></span> (bootloader) is-logical:mdcompress:no</span></span> (bootloader) is-logical:mdtp_a:no</span></span> (bootloader) is-logical:mdtp_b:no</span></span> (bootloader) is-logical:mdtpsecapp_a:no</span></span> (bootloader) is-logical:mdtpsecapp_b:no</span></span> (bootloader) is-logical:metadata:no</span></span> (bootloader) is-logical:misc:no</span></span> (bootloader) is-logical:modem_a:no</span></span> (bootloader) is-logical:modem_b:no</span></span> (bootloader) is-logical:modemdump:no</span></span> (bootloader) is-logical:modemst1:no</span></span> (bootloader) is-logical:modemst2:no</span></span> (bootloader) is-logical:multiimgoem_a:no</span></span> (bootloader) is-logical:multiimgoem_b:no</span></span> (bootloader) is-logical:ocdt:no</span></span> (bootloader) is-logical:odm_b-cow:yes</span></span> (bootloader) is-logical:odm_b:yes</span></span> (bootloader) is-logical:oplusdycnvbk:no</span></span> (bootloader) is-logical:opluslog:no</span></span> (bootloader) is-logical:oplusreserve1:no</span></span> (bootloader) is-logical:oplusreserve2:no</span></span> (bootloader) is-logical:oplusreserve3:no</span></span> (bootloader) is-logical:oplusreserve4:no</span></span> (bootloader) is-logical:oplusreserve5:no</span></span> (bootloader) is-logical:oplus_sec_a:no</span></span> (bootloader) is-logical:oplus_sec_b:no</span></span> (bootloader) is-logical:oplusstanvbk_a:no</span></span> (bootloader) is-logical:oplusstanvbk_b:no</span></span> (bootloader) is-logical:param:no</span></span> (bootloader) is-logical:persist_bkp:no</span></span> (bootloader) is-logical:persist:no</span></span> (bootloader) is-logical:product_b-cow:yes</span></span> (bootloader) is-logical:product_b:yes</span></span> (bootloader) is-logical:qmcs:no</span></span> (bootloader) is-logical:qsee_log:no</span></span> (bootloader) is-logical:qupfw_a:no</span></span> (bootloader) is-logical:qupfw_b:no</span></span> (bootloader) is-logical:qweslicstore_a:no</span></span> (bootloader) is-logical:qweslicstore_b:no</span></span> (bootloader) is-logical:rawdump:no</span></span> (bootloader) is-logical:rtice:no</span></span> (bootloader) is-logical:sda:no</span></span> (bootloader) is-logical:sdb:no</span></span> (bootloader) is-logical:sdc:no</span></span> (bootloader) is-logical:sdd:no</span></span> (bootloader) is-logical:sde:no</span></span> (bootloader) is-logical:sdf:no</span></span> (bootloader) is-logical:secdata:no</span></span> (bootloader) is-logical:shrm_a:no</span></span> (bootloader) is-logical:shrm_b:no</span></span> (bootloader) is-logical:splash_a:no</span></span> (bootloader) is-logical:splash_b:no</span></span> (bootloader) is-logical:spunvm:no</span></span> (bootloader) is-logical:ssd:no</span></span> (bootloader) is-logical:storsec:no</span></span> (bootloader) is-logical:super:no</span></span> (bootloader) is-logical:system_b-cow:yes</span></span> (bootloader) is-logical:system_b:yes</span></span> (bootloader) is-logical:system_ext_b-cow:yes</span></span> (bootloader) is-logical:system_ext_b:yes</span></span> (bootloader) is-logical:tz_a:no</span></span> (bootloader) is-logical:tz_b:no</span></span> (bootloader) is-logical:tzsc:no</span></span> (bootloader) is-logical:uefisecapp_a:no</span></span> (bootloader) is-logical:uefisecapp_b:no</span></span> (bootloader) is-logical:uefivarstore:no</span></span> (bootloader) is-logical:userdata:no</span></span> (bootloader) is-logical:vbmeta_a:no</span></span> (bootloader) is-logical:vbmeta_b:no</span></span> (bootloader) is-logical:vbmeta_system_a:no</span></span> (bootloader) is-logical:vbmeta_system_b:no</span></span> (bootloader) is-logical:vbmeta_vendor_a:no</span></span> (bootloader) is-logical:vbmeta_vendor_b:no</span></span> (bootloader) is-logical:vendor_b-cow:yes</span></span> (bootloader) is-logical:vendor_boot_a:no</span></span> (bootloader) is-logical:vendor_boot_b:no</span></span> (bootloader) is-logical:vendor_b:yes</span></span> (bootloader) is-logical:vm-bootsys_a:no</span></span> (bootloader) is-logical:vm-bootsys_b:no</span></span> (bootloader) is-logical:vm-data:no</span></span> (bootloader) is-logical:xbl_a:no</span></span> (bootloader) is-logical:xbl_b:no</span></span> (bootloader) is-logical:xbl_config_a:no</span></span> (bootloader) is-logical:xbl_config_b:no</span></span> (bootloader) is-userspace:yes</span></span> (bootloader) max-download-size:0x10000000</span></span> (bootloader) partition-size:abl_a:0x800000</span></span> (bootloader) partition-size:abl_b:0x800000</span></span> (bootloader) partition-size:abl_log:0xFD0000</span></span> (bootloader) partition-size:ALIGN_TO_128K_1:0x1A000</span></span> (bootloader) partition-size:ALIGN_TO_128K_2:0x1A000</span></span> (bootloader) partition-size:android_log:0xFD0000</span></span> (bootloader) partition-size:aop_a:0x80000</span></span> (bootloader) partition-size:aop_b:0x80000</span></span> (bootloader) partition-size:apdp:0x40000</span></span> (bootloader) partition-size:apdp_full:0x40000</span></span> (bootloader) partition-size:bluetooth_a:0x200000</span></span> (bootloader) partition-size:bluetooth_b:0x200000</span></span> (bootloader) partition-size:boot_a:0xC000000</span></span> (bootloader) partition-size:boot_b:0xC000000</span></span> (bootloader) partition-size:carrier:0x4000000</span></span> (bootloader) partition-size:cdt:0x20000</span></span> (bootloader) partition-size:connsec:0x20000</span></span> (bootloader) partition-size:cpucp_a:0x100000</span></span> (bootloader) partition-size:cpucp_b:0x100000</span></span> (bootloader) partition-size:ddr:0x100000</span></span> (bootloader) partition-size:devcfg_a:0x20000</span></span> (bootloader) partition-size:devcfg_b:0x20000</span></span> (bootloader) partition-size:devinfo:0x1000</span></span> (bootloader) partition-size:dinfo:0x4000</span></span> (bootloader) partition-size:dip:0x100000</span></span> (bootloader) partition-size:DRIVER:0x2000000</span></span> (bootloader) partition-size:dsp_a:0x4000000</span></span> (bootloader) partition-size:dsp_b:0x4000000</span></span> (bootloader) partition-size:dtbo_a:0x1800000</span></span> (bootloader) partition-size:dtbo_b:0x1800000</span></span> (bootloader) partition-size:engineering_cdt_a:0x100000</span></span> (bootloader) partition-size:engineering_cdt_b:0x100000</span></span> (bootloader) partition-size:featenabler_a:0x20000</span></span> (bootloader) partition-size:featenabler_b:0x20000</span></span> (bootloader) partition-size:frp:0x80000</span></span> (bootloader) partition-size:fsc:0x20000</span></span> (bootloader) partition-size:fsg:0x300000</span></span> (bootloader) partition-size:hyp_a:0x800000</span></span> (bootloader) partition-size:hyp_b:0x800000</span></span> (bootloader) partition-size:hyp_log:0xFD0000</span></span> (bootloader) partition-size:imagefv_a:0x200000</span></span> (bootloader) partition-size:imagefv_b:0x200000</span></span> (bootloader) partition-size:kernel_log:0xFD0000</span></span> (bootloader) partition-size:keymaster_a:0x80000</span></span> (bootloader) partition-size:keymaster_b:0x80000</span></span> (bootloader) partition-size:keystore:0x80000</span></span> (bootloader) partition-size:limits:0x1000</span></span> (bootloader) partition-size:limits-cdsp:0x1000</span></span> (bootloader) partition-size:logdump:0x4000000</span></span> (bootloader) partition-size:logfs:0x800000</span></span> (bootloader) partition-size:mdcompress:0x1400000</span></span> (bootloader) partition-size:mdtp_a:0x2000000</span></span> (bootloader) partition-size:mdtp_b:0x2000000</span></span> (bootloader) partition-size:mdtpsecapp_a:0x400000</span></span> (bootloader) partition-size:mdtpsecapp_b:0x400000</span></span> (bootloader) partition-size:metadata:0x1000000</span></span> (bootloader) partition-size:misc:0x100000</span></span> (bootloader) partition-size:modem_a:0x14A00000</span></span> (bootloader) partition-size:modem_b:0x14A00000</span></span> (bootloader) partition-size:modemdump:0x12C00000</span></span> (bootloader) partition-size:modemst1:0x300000</span></span> (bootloader) partition-size:modemst2:0x300000</span></span> (bootloader) partition-size:multiimgoem_a:0x8000</span></span> (bootloader) partition-size:multiimgoem_b:0x8000</span></span> (bootloader) partition-size:ocdt:0x20000</span></span> (bootloader) partition-size:odm_b:0x2BD000</span></span> (bootloader) partition-size:odm_b-cow:0x2C1000</span></span> (bootloader) partition-size:oplusdycnvbk:0xA00000</span></span> (bootloader) partition-size:opluslog:0x10000000</span></span> (bootloader) partition-size:oplusreserve1:0x800000</span></span> (bootloader) partition-size:oplusreserve2:0x10000000</span></span> (bootloader) partition-size:oplusreserve3:0x4000000</span></span> (bootloader) partition-size:oplusreserve4:0x2000000</span></span> (bootloader) partition-size:oplusreserve5:0x4000000</span></span> (bootloader) partition-size:oplus_sec_a:0xA00000</span></span> (bootloader) partition-size:oplus_sec_b:0xA00000</span></span> (bootloader) partition-size:oplusstanvbk_a:0xA00000</span></span> (bootloader) partition-size:oplusstanvbk_b:0xA00000</span></span> (bootloader) partition-size:param:0x800000</span></span> (bootloader) partition-size:persist:0x2000000</span></span> (bootloader) partition-size:persist_bkp:0x2000000</span></span> (bootloader) partition-size:product_b:0x1652F000</span></span> (bootloader) partition-size:product_b-cow:0x16696000</span></span> (bootloader) partition-size:qmcs:0x1E00000</span></span> (bootloader) partition-size:qsee_log:0xFD0000</span></span> (bootloader) partition-size:qupfw_a:0x14000</span></span> (bootloader) partition-size:qupfw_b:0x14000</span></span> (bootloader) partition-size:qweslicstore_a:0x40000</span></span> (bootloader) partition-size:qweslicstore_b:0x40000</span></span> (bootloader) partition-size:rawdump:0x8000000</span></span> (bootloader) partition-size:rtice:0x80000</span></span> (bootloader) partition-size:sda:0x3A10800000</span></span> (bootloader) partition-size:sdb:0x800000</span></span> (bootloader) partition-size:sdc:0x800000</span></span> (bootloader) partition-size:sdd:0x2000000</span></span> (bootloader) partition-size:sde:0x180000000</span></span> (bootloader) partition-size:sdf:0x2000000</span></span> (bootloader) partition-size:secdata:0x7000</span></span> (bootloader) partition-size:shrm_a:0x20000</span></span> (bootloader) partition-size:shrm_b:0x20000</span></span> (bootloader) partition-size:splash_a:0x20A4000</span></span> (bootloader) partition-size:splash_b:0x20A4000</span></span> (bootloader) partition-size:spunvm:0x2000000</span></span> (bootloader) partition-size:ssd:0x2000</span></span> (bootloader) partition-size:storsec:0x20000</span></span> (bootloader) partition-size:super:0x29B000000</span></span> (bootloader) partition-size:system_b:0x406A4000</span></span> (bootloader) partition-size:system_b-cow:0x40AAC000</span></span> (bootloader) partition-size:system_ext_b:0xDF3A000</span></span> (bootloader) partition-size:system_ext_b-cow:0xE01B000</span></span> (bootloader) partition-size:tz_a:0x400000</span></span> (bootloader) partition-size:tz_b:0x400000</span></span> (bootloader) partition-size:tzsc:0x20000</span></span> (bootloader) partition-size:uefisecapp_a:0x200000</span></span> (bootloader) partition-size:uefisecapp_b:0x200000</span></span> (bootloader) partition-size:uefivarstore:0x80000</span></span> (bootloader) partition-size:userdata:0x3751FB3000</span></span> (bootloader) partition-size:vbmeta_a:0x10000</span></span> (bootloader) partition-size:vbmeta_b:0x10000</span></span> (bootloader) partition-size:vbmeta_system_a:0x10000</span></span> (bootloader) partition-size:vbmeta_system_b:0x10000</span></span> (bootloader) partition-size:vbmeta_vendor_a:0x10000</span></span> (bootloader) partition-size:vbmeta_vendor_b:0x10000</span></span> (bootloader) partition-size:vendor_b:0x50520000</span></span> (bootloader) partition-size:vendor_b-cow:0x50A27000</span></span> (bootloader) partition-size:vendor_boot_a:0xC000000</span></span> (bootloader) partition-size:vendor_boot_b:0xC000000</span></span> (bootloader) partition-size:vm-bootsys_a:0x10625000</span></span> (bootloader) partition-size:vm-bootsys_b:0x10625000</span></span> (bootloader) partition-size:vm-data:0x20A4000</span></span> (bootloader) partition-size:xbl_a:0x600000</span></span> (bootloader) partition-size:xbl_b:0x600000</span></span> (bootloader) partition-size:xbl_config_a:0x80000</span></span> (bootloader) partition-size:xbl_config_b:0x80000</span></span> (bootloader) product:OnePlus9Pro</span></span> (bootloader) secure:yes</span></span> (bootloader) security-patch-level:2021-12-05</span></span> (bootloader) serialno:XXXXXXX</span></span> (bootloader) slot-count:2</span></span> (bootloader) snapshot-update-status:snapshotted</span></span> (bootloader) super-partition-name:super</span></span> (bootloader) system-fingerprint:OnePlus/OnePlus9Pro_EEA/OnePlus9Pro:11/RKQ1.201105.002/2111112053:user/release-keys</span></span> (bootloader) treble-enabled:true</span></span> (bootloader) unlocked:yes</span></span> (bootloader) vendor-fingerprint:OnePlus/OnePlus9Pro_EEA/OnePlus9Pro:11/RKQ1.201105.002/2111112053:user/release-keys</span></span> (bootloader) version:0.4</span></span> (bootloader) version-baseband:</span></span> (bootloader) version-bootloader:unknown</span></span> (bootloader) version-os:11</span></span> (bootloader) version-vndk:30</span></span> Finished. Total time: 1.181s</span></span></code></pre> Fuck. Again. ... TBC. AGAIN. how many more yaks are left to shave one wonders.</p> todo:</p> skip all the guided setup steps</li> open the microG app</li> run the self check</li> magisk root if needed</li> everything else</li> </ul> config</h3> settings sound</li> advanced screen locking sound - off</li> touch sounds - off</li> </ul> </li> system</li> advanced gestures power menu advanced restart - on</li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> Lineage+microG+Magisk again - success!</h3> Let's try downloading the latest and flashing again.</p> https://download.lineage.microg.org/lemonadep/?sort=time&order=asc</a></p> tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span> $ ll lineage-18.1-2022040</span></span> -rw-rw-r-- 1 tim tim 192M Apr 30 20:38 lineage-18.1-20220401-microG-lemonadep-recovery.img</span></span> -rw-rw-r-- 1 tim tim 1.1G Apr 30 20:39 lineage-18.1-20220405-microG-lemonadep.zip</span></span> -rw-rw-r-- 1 tim tim 3.9K Apr 30 20:37 lineage-18.1-20220405-microG-lemonadep.zip.prop</span></span> -rw-rw-r-- 1 tim tim 109 Apr 30 20:37 lineage-18.1-20220405-microG-lemonadep.zip.sha256sum</span></span> $ sha256sum -c lineage-18.1-20220405-microG-lemonadep.zip.sha256sum</span></span> lineage-18.1-20220405-microG-lemonadep.zip: OK</span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span> $ cd update_verifier-master </span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/update_verifier-master</span></span> $ python update_verifier.py lineageos4microg_pubkey ../lineage-18.1-20220405-microG-lemonadep.zip </span></span> verified successfully</span></span></code></pre> We'll do a swap slot again so that it won't overwrite the currently working lineage install.</p> Vol-down + power button to enter recovery (from powered off).</p> Lineage recovery says "Active slot: b" at the top. Let's boot it, check what we have, swap, boot again to see the other...</p> slot b: broken install - drops to bootloader</li> slot a: also broken ...., erm I swear I booted that. wtf. phones.</li> </ul> so that means I should swap slot so that "a" is active, so that fastboot flashes over the broken "b" and then swaps to it without touching the currently working "a" slot.</p> Use lineage bootloader to format the system just to see if that fixes the broken install. .... Nope. Both slots still fubar.</p> Right, well, flashing time either way.</p> put phone into recover "update > apply from adb" mode</p> adb sideload lineage-18.1-20220405-microG-lemonadep.zip</code></p> long pause...</p> back > reboot system</p> hurrah, a lineage boot animation</p> rattle through setup steps again, did wifi, nothing else</p> todo: fingerprint and pin</p> todo: re-root with magisk before doing anything else</p> new things compared to plain lineage: f-droid & microG apps</p> installed andOTP & vespucci (OSM editor) from f-droid</p> enable developer options (system > about phone, tap lots of times)</p> system > advanced > developer options > usb debugging ON + Disable adb authorisation timeout (so that if my screen fails or the software misbehaves I can get my files over a usb cable)</p> Install Magisk by John Wu via F-Droid instead of adb this time</p> Extract the updated payload:</p> tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span> $ cd payload </span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ ../../payload-dumper-go/payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go ../lineage-18.1-20220405-microG-lemonadep.zip</span></span> Please wait while extracting payload.bin from the archive.</span></span> payload.bin: /tmp/payload_208101762.bin</span></span> Payload Version: 2</span></span> Payload Manifest Length: 94295</span></span> Payload Manifest Signature Length: 267</span></span> Found partitions:</span></span> boot (201 MB), dtbo (25 MB), odm (2.9 MB), product (378 MB), system (1.1 GB), system_ext (234 MB), vbmeta (8.2 kB), vbmeta_system (4.1 kB), vendor (1.3 GB), vendor_boot (201 MB)</span></span> Number of workers: 4</span></span> odm (2.9 MB) [==============================================================================================] 100 %</span></span> boot (201 MB) [==============================================================================================] 100 %</span></span> dtbo (25 MB) [==============================================================================================] 100 %</span></span> product (378 MB) [==============================================================================================] 100 %</span></span> system (1.1 GB) [==============================================================================================] 100 %</span></span> system_ext (234 MB) [==============================================================================================] 100 %</span></span> vbmeta (8.2 kB) [==============================================================================================] 100 %</span></span> vbmeta_system (4.1 kB) [==============================================================================================] 100 %</span></span> vendor (1.3 GB) [==============================================================================================] 100 %</span></span> vendor_boot (201 MB) [==============================================================================================] 100 %</span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ ll</span></span> total 4.0K</span></span> drwxr-xr-x 2 tim tim 4.0K Apr 30 21:34 extracted_20220430_213414</span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ ll extracted_20220430_213414 </span></span> total 3.3G</span></span> -rwxr-xr-x 1 tim tim 192M Apr 30 21:34 boot.img</span></span> -rwxr-xr-x 1 tim tim 24M Apr 30 21:34 dtbo.img</span></span> -rwxr-xr-x 1 tim tim 2.8M Apr 30 21:34 odm.img</span></span> -rwxr-xr-x 1 tim tim 361M Apr 30 21:34 product.img</span></span> -rwxr-xr-x 1 tim tim 224M Apr 30 21:34 system_ext.img</span></span> -rwxr-xr-x 1 tim tim 1.1G Apr 30 21:34 system.img</span></span> -rwxr-xr-x 1 tim tim 8.0K Apr 30 21:34 vbmeta.img</span></span> -rwxr-xr-x 1 tim tim 4.0K Apr 30 21:34 vbmeta_system.img</span></span> -rwxr-xr-x 1 tim tim 192M Apr 30 21:34 vendor_boot.img</span></span> -rwxr-xr-x 1 tim tim 1.3G Apr 30 21:35 vendor.img</span></span></code></pre> send to phone:</p> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ adb push extracted_20220430_213414/boot.img /sdcard/Download/</span></span> extracted_20220430_213414/boot.img: 1 file pushed, 0 skipped. 192.7 MB/s (201326592 bytes in 0.996s)</span></span></code></pre> run the patch in the phone Magisk UI (under “install”)</p> get the patched file back:</p> im@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ patched=`adb shell ls /sdcard/Download/magisk_patched`</span></span> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ adb pull $patched</span></span> /sdcard/Download/magisk_patched-24300_v4xhs.img: 1 file pulled, 0 skipped. 34.7 MB/s (201326592 bytes in 5.532s)</span></span></code></pre> turn on advanced reboot again in settings</p> reboot to fastboot</p> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span> $ fastboot flash boot magisk_patched-24300_v4xhs.img </span></span> Sending 'boot_a' (196608 KB) OKAY [ 6.597s]</span></span> Writing 'boot_a' OKAY [ 0.611s]</span></span> Finished. Total time: 7.387s</span></span></code></pre> "reboot now"</p> success. open up magisk, shows as installed (v24.3)</p> F-Droid app list</h2> Turn On Install history</h3> Before starting installations do this so that we (might) have an exportable record to share.</p> Turn on install history before starting so that we can export at the end - https://gitlab.com/fdroid/fdroidclient/-/issues/70#note_89110104</a></p> Settings > Expert mode > Keep install history</p> Export list when done</h3> Settings > Install history and metrics > Share (share icon with 3 linked dots at top) > fail, no apps that know what to do with the file installed yet. gah.</p> Google contacts, calendar & gmail - fail</h3> via aurora</p> Install contacts...</p> Sign in</li> Goes to microG</li> Then to official google sign in</li> Sign in succeeds</li> Acc now listed in settings > accounts</li> google contacts still shows no accounts & no contacts. Damn</li> tap acc, tap any of the settings (all go to same place) turn on "Allow apps to find contacts"</li> now google contacts shows the account, but still doesn't sync.</li> </ul> Hrmm. Need this or equivalent obviously. Switching will be non trivial.</p> Same problem for gmail.</p> Calendar app works and has all events, desktop widget broken (shows no events)</p> Research</h4> Alternatives</h5> https://alternativeto.net/software/google-contacts/</a></li> https://www.reddit.com/r/LineageOS/comments/7wnwpe/alternative_to_google_calender_or_contacts_sync/</a></li> https://www.reddit.com/r/LineageOS/comments/8qv3p3/alternative_services_to_google_calendar_and/</a></li> https://www.reddit.com/r/LineageOS/comments/pt5ce7/are_the_contacts_and_telephone_apps_built_by/</a> https://f-droid.org/en/packages/com.simplemobiletools.contacts.pro/</a></li> </ul> </li> https://f-droid.org/en/packages/com.simplemobiletools.contacts.pro/</a></li> </ul> Making it work</h5> https://github.com/microg/GmsCore/issues/585</a></li> https://github.com/tomwiggers/Magisk-ConCal-Sync-MicroG</a> download Magisk-ConCal-Sync-MicroG repo zip</a></li> extract, go into inner folder and zip up the four items</li> copy to phone with syncthing</li> magisk > modules > install from storage > select the zip > done</li> reboot phone</li> boot loop. FUUUUUUUUCK</li> reported issue for what it's worth, including logcat</code> logs: https://github.com/tomwiggers/Magisk-ConCal-Sync-MicroG/issues/3</a></li> remove modules https://www.thecustomdroid.com/uninstall-magisk-modules-in-twrp-recovery-guide/</a></li> connect phone usb</li> run adb wait-for-device shell magisk --remove-modules</code> on laptop</li> boot up phone normally</li> phone booted into lock screen, phew!</li> </ul> </li> </ul> </li> </ul> Calendar</h4> bearable, widget broken but app itself works and syncs</p> have to manually turn on sync for additional calendars</p> Contacts</h4> Need alternative, perhaps fastmail</p> Let's try fastmail's contacts... google contacts web app</a> > export from google contacts > vCard</li> fastmail web app > dropdown thing > settings</a> > import > contacts > upload addressbook file > choose the vcf imported all but two, showed me the errors, not ones I care about.</li> </ul> </li> check fastmail phone app, contacts come flooding in.</li> but no sign of them in the main phone contacts app. damn.</li> </ul> </li> https://www.reddit.com/r/fastmail/comments/f2akqj/dealing_with_contacts/</a></li> </ul> I've messaged fastmail support to ask them why they have a completely separate contact section in the fastmail app rather than using the shared contacts store in Android/LineageOS.</p> Because I'm short of good options I will reluctantly try the DAVx app that everyone points to. I really don't want yet another sync involved in the whole thing but I also don't want to run and maintain a NextCloud install (which lots of people suggest).</p> Installing DAVx5 It's listed in Aurora (the play store proxy) and shows "£4.29" instead of the usual "Install".</li> Pressed money button to see what happens > error "Download failed, app not purchased". Cool.</li> Go to DAVx5 on play store</a> and make the purchase concerningly for the future the play store pops an "install to blah device", which is fine now because it knows about my old</em> phone, but what if I didn't have the old phone some time, does that mean I could no longer get paid apps and install them with Aurora?</li> </ul> </li> Back on the new phone Aurora install now works. Win.</li> </ul> </li> Setting up DAVx5 rattled through default wizard without changing (or understanding) anything</li> interesting warning about oneplus blocking sync that points to https://www.davx5.com/faq/synchronization-is-not-run-as-expected</a></li> Create app password in fastmail web interface for DAVx</li> Login to DAVx app with email + app-password ... fail, presumably because it's a custom domain</li> Use custom base url option - https://www.davx5.com/tested-with/fastmail</a> Base url: https://carddav.fastmail.com</code> + email + app-password .... success</li> Create account: account name? (defaults to tim@timwise.co.uk) - "Use your email address as account name because Android will use the account name as ORGANIZER field for events you create. You can't have two accounts with the same name." - probably fine, left default</li> Contact group method: "Groups are separate vCards" - default, left as this</li> "Groups are per-contact categories"</li> </ul> </li> </ul> </li> Next</li> No contacts sync (missing permissions) > Permissions > All of the below to "on" > grant dialogs</li> Shows two unchecked checkboxes, but don't know what they mean "Global Address Book"</li> "personal"</li> </ul> </li> Trying out ticking "personal", tick it and press orange sync now button</li> woo, main contacts app now full of contacts. sim switch time!!!</li> </ul> </li> </ul> GMail</h4> todo, probably set up imap and use k9mail + web interface on laptop</p> F-Droid app list</h3> Vespucci (open-streetmap (OSM) editor)</li> andOTP (one time password generator with backup/restore capability)</li> Aurora play store proxy - configure as per above section</li> Syncthing</li> OAndBackupX - to replace proprietary (but very good) titanium backup Needs root, provided by Magisk. Without root no apps can access the data of other apps. This is the primary reason that root is so important to me. I want a backup system that I can actually trust because it's not some proprietary cloud magic pixie dust, it's instead a straight-forward grab-the-files-and-put-them-somewhere-safe operation that I can inspect and get to without more proprietary black magic.</li> </ul> </li> KeepassDX</a> - https://github.com/Kunzisoft/KeePassDX</a></li> k9mail</li> vlc</li> markor</li> antennapod - todo restore state</li> qr & barcode scanner</li> aTimeLogger</li> Editor</a> (replacement for droidedit?)</li> GPSLogger</a></li> </ul> todo - more apps (f-droid where possible)</h3> VLC</li> AntennaPod</li> Audible</li> Maverick (for recording and viewing gps traces, could do with a replacement, the map tiles seem to have died and it's getting buggy)</li> google home (for chromecast & google mini integration)</li> RainAlarm (paid for ad-free)</li> Google calendar (or find a non-google replacement)</li> Google contacts (or find a non-google replacement)</li> Waze</li> Google maps (for satnav, route planning and finding local businesses)</li> LinkedIn</li> Twitter</li> Telegram FOSS (a slightly free-er version of telegram)</li> ...</li> </ul> Other f-droid app suggestions</h3> https://www.reddit.com/r/fdroid/comments/lzzdbg/a_list_of_some_fdroid_apps_i_installed_to_get_you/</a></li> </ul> Play store app list (via Aurora)</h2> Only things not available in f-droid</p> maps.me</li> Spotify</li> Audible</li> Firefox</li> google maps - works flawlessly!</li> signal messemger surprisingly not in f-droid!</li> can't be on multiple phones at once, can only "transfer". Grumble.</li> </ul> </li> banking apps (2/3 work)</li> sunsama</li> trello</li> slack</li> rain alarm - todo restore paid version</li> zoom</li> bulb energy</li> YNAB</li> Ring (doorbell)</li> Google home</li> Life360 (shared gps locations for family, saves a bunch of "have you left yet?" text messages)</li> Fastmail - not for the mail, prefer k9mail, but to see if it can do contacts as well as goooooooogle</li> DroidEdit</li> Ringo</li> National rail</li> Goodreads</li> Pluralsight</li> LinkedIn</li> FreeAgent</li> BBC Weather</li> Sounds</li> iPlayer</li> </ul> </li> Amazon Shopping</li> Amazon Prime</li> YouTube</li> Microsoft authenticator</li> Google authenticator</li> Bandcamp</li> Offline English Dictionary by Livio</a></li> Strava</li> </ul> Built-in apps</h2> Camera - configure</h3> Hamburger GPS > on</li> Touch focus duration > infinite (man that has always annoyed me! Default 3 seconds, why?!)</li> shutter sound > disabled</li> </ul> </li> </ul> Browser - disable</h3> Disable (using firefox instead)</p> Messaging - disable</h3> Messaging > long press > app info > disable</p> Individual app setup</h2> Syncthing</h3> Open web gui</li> Disable all non-LAN</li> Remove default folder (Camera)</li> Set device name</li> Connect to existing machine(s).</li> Add /storage/emulated/0</code> folder (can't do this in non-web gui, stupid security warning. Nanny state of android.) Warnings about failure to sync Android/data</code> and Android/obb</code>. Looks like they've tightened up security more, either that or syncthing has better warnings now. These folders will be exposed by OAndBackupX so I can safely ignore this until I sort out the ignore file for syncthing.</li> </ul> </li> </ul> Oh no, syncthing can no longer access /storage/emulated/0/Android/data</code> - https://forum.syncthing.net/t/can-i-sync-send-only-data-from-android-data-in-android-11/17898</a></p> Even more reason to have root and run a real and independent backup tool.</p> Syncthing ignore patterns</h4> Set up ignores for caches and things that won't sync. Files for this are on github at https://github.com/timabell/dotmatrix/tree/master/phone</a></p> Create file phone.stignore</code> in the root of the sync'd sdcard folder. This one you can edit on the laptop as you find more things you don't want sync'd.</li> Create file phonetemplate.stignore</code> in the root folder and put only this line in it #include phone.stignore</code></li> On the phone when they have sync'd copy the phonetemplate.stignore</code> to a file named just .stignore</code> in the root folder.</li> </ul> The reason for this dance is that syncthing won't sync the .stignore</code> so to be able to edit the ignores on a different device you have to set up a link.</p> AndOTP</h3> Set up password</li> Turn off global timeout bar (distracting & stressful)</li> Icons: small</li> Layout: compact</li> Copy backup from old phone with syncthing and restore it</li> </ul> AntennaPod</h3> Now I get to the crux of why I was so fucked off with Android.</p> I want to copy the exact</em> state of antennapod to the new phone; but my old phone de-rooted itself and broke the backups; and you can't back up ~/data where everything lives without root because it's all locked down with per-app-user permissions, which is great for isolating apps from each other ... and you. The only option for those folders is in app sync (e.g.. sign in to trello/firefox/slack etc) so local state is throwaway, or backup to google drive .... to some magical fucking invisible folder so you can't even download the backups it made. This shit makes me so mad.</p> We already have working systems for backups and security and these fuckers have broken them and replaced them with their own vendor-locked-in proprietary black-box shite that doesn't even work all the time.</p> Because I'd lost root and usable backups it took me a while to work out how to transfer my latest beloved podcast player state to a new phone. I knew that you could import/export an OPML list of suscriptions but I really wanted to keep the play state of every episode so I caould easily scan the back-catalogues for episodes I'd never listened to dotted amongst the many thousands of episodes I have listned to. Well darm it there is a full export feature I didn't know about. It exports to the virtual sdcard storagea which is kinda like a home directory that you can actually access as a user. I already have the sdcard storage syncing to my other machines with syncthing so getting it from there onwards is a trivial copy-paste on the laptop.</p> Here's what a quick google</del> ddg</a> showed up:</p> https://github.com/AntennaPod/AntennaPod/issues/5391</a></li> https://github.com/AntennaPod/AntennaPod/issues/5031</a></li> https://github.com/AntennaPod/AntennaPod/issues/377</a></li> </ul> Old phone:</p> hamburger menu top left</li> settings (cog at bottom) > storage > import/export > database export</li> ensure syncthing is running and connected</li> </ul> Laptop:</p> Copy file on laptop to new phone's sync folder</li> </ul> New phone:</p> make sure symcthing is running, connected and up to date to pull new file in</li> hamburger menu top left</li> settings (cog at bottom) > storage > import/export > database import > Confirm</li> select file</li> press ok to restart Antennapod</li> Woo! Queue entirely intact!</li> </ul> Downloads have to be re-done for the queue but the UI makes that dead easy, just hit play and it switches to a download button so you can whip through them quick.</p> Bulb energy</h3> App downloaded</li> Sign in by magic email link failed</li> Sign in with password worked</li> Statement download failed with "no apps can perform this action" but can always use website to get them. ... ah, no pdf reading apps installed, installing a random pdf app fixed that. Not a very clear error message that one!</li> </ul> KeePassDX</h3> Enable fingerprint unlock https://github.com/Kunzisoft/KeePassDX/wiki/Advanced-Unlocking#b-link-database-credential-to-advanced-unlock-recognition=</a></p> NeoBackup</h3> Backup folder > neo-backup folder on emulated sdcard home > use this folder</li> Allow access prompt > allow</li> </ul> "i" button top right > interesting note about types of android data and inability to restore apps using hardware key stores such as signal</p> work out what apps to back up -- all of them, why guess what I'll need with so much storage available</p> </li> actual backups - done</p> </li> schedule backups - done, works great</p> </li> maybe automate clearing off phone onto laptop/server</p> </li> Set backup password</p> </li> </ul> Set scheduled backup:</p> 7 days</li> 5am</li> system apps + user apps</li> apk + obb + data + device protected data</li> all apps</li> enabled</li> Start now</li> </ul> Firefox</h3> 3dots > settings > Search > duckduckgo</p> firefox was active for 6 hours which makes me think it isn't backgrounding properly, perhaps because of media or js on websites.</p> Firefox icon > long press > App info (i) > Advanced > Battery > Background restriction > Restrict</p> Signal</h3> Use transfer account to new phone feature Enable chat backups to virtual SD: dots > settings > chat. > Chat backups. Set folder and create backup.</p> Signal popup saying detected issue with play services and establishing own connection. Now have permanent notification.</p> GPSLogger</h3> https://f-droid.org/en/packages/com.mendhak.gpslogger/</a></p> hamburger logging details save to folder > give permissions > new folder (+) > ~/gpx</code></li> New file creation > every time I start</li> </ul> </li> performance Logging interval > 5 secs</li> </ul> </li> </ul> </li> </ul> So that syncthing can see it. Defaults to inaccessible data folder. Fffffff</p> Also set to create new track every time.</p> A thought on moving apps and data between phones</h2> During the 1000th discussion of phones with my long-sufferring better-half I happened to utter a line that I think realy captures why this whole nonsense is (probably) worth the effort:</p> "Living in the iOS ecosystem with their magical one-tap transfer from old phones to new phones that leave the new phone exactly in the same state as your old phone ... until you realize one day you want to leave the ecosystem, and you have to leave all your data behind in their walled garden." ~ Me</p> </blockquote> There are several ways of moving data between phones, which provide varying levels of protection against data loss disasters. Disasters that could be local (phone down loo), localized (all devices stolen from house), cloud based (cloud account hacked and wiped), a combination (cloud hacked and devices remote-wiped).</p> Copying files directly</h3> Something like syncthing.</p> Only works for data on virtual sdcard such as photos & downloads.</p> Local backup/restore</h3> With OAndBackup, NeoBackup, TitaniumBackup etc.</p> Needs root.</p> Can get app's private data out to storage you control.</p> Can also backup set of installed apps (.apk</code>s) which it turns out is actually quite a time saver compared to clicking install in the app store 100+ times.</p> Vendor-provided cloud backup</h3> Completely walled-garden solution. No way to get your data off their cloud without restoring to another device they support (i.e. another Android/iOS device).</p> 3rd party cloud backup</h3> Something like dropbox.</p> It seems that the duopoly has deemed this unacceptable so have hobbled these apps fom working properly or at all. Dropbox on iOS has been reduced to a glorified photo backup which can't even run in the background.</p> Personally I think the fact that the duopoly is denying competition is anti-competitive practice the likes of which saw Microsoft get record fines from the EU back in the day. Hopefully the same will happen to Apple & Google to crack this open a bit, but until then we'll have to just shovel digital shit like this blog post to get out from under them.</p> File transfer</h2> Easy thanks to syncthing, just move the folders & files on the laptop from sync sdcard folder of the old phone to the one for the new phone. Win.</p> Trial TitaniumBackup restore - success</h2> Before my old phone de-rooted itself thanks to a F***g OS update (grrrrr) I did have titanium set up, so I have somem rather stale backups from before then. Mostly as an experiment I've installed Titanium on the new phone to see if the restore works. This is the first time I've had two capable and still working phones at the same time to be able to experiment like this.</p> Installed TitaniumBackup via Aurora.</li> Installed Titanium's "key" app via Aurora.</li> Titanium complained that Play store wasn't responding so couldn't check licence key. Fucksake. More lockin.</li> Removed key app again.</li> Copied one of the old backups across (google authenticator). (An app I had not yet installed on the new phone)</li> Fired up titanium, hit restore for that app.</li> Boom! There's the app installed, fired it up, worked no problems and now have an (outdated) list of OTPs that I don't use any more. Hollow victory achieved.</li> </ul> Clock</h2> Settings Time in seconds > on</li> Alarms silence after > 5 mins</li> Snooze length > 15 minutes - I find that if I'm really too tired to move 15 mins makes a good power nap but isn't long enough to be disastrous</li> gradually increase</li> vol button to snooze</li> </ul> </li> </ul> </li> </ul> Config</h2> A complete list</p> Settings:</p> Network and Internet Hotspot and Tethering Wi-Fi Hotspot On</li> AP Band 5 GHz band preferred</li> </ul> </li> </ul> </li> Bluetooth tethering > On (to give the laptop internet out and about)</li> </ul> </li> Connected Devices Connection preferences Bluetooth Device name: foo</li> </ul> </li> </ul> </li> </ul> </li> Wifi Wifi preferences Notify for public networks > off</li> </ul> </li> </ul> </li> </ul> </li> Apps and Notifications Default apps SMS app > Signal (actually a great integrated SMS app)</li> </ul> </li> Notifications Advanced Do Not Disturb People Calls > "Contacts and repeat callers"</li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> </ul> </li> Battery Battery saver and performance Automatic power saver - 15% (off by default oddly)</li> </ul> </li> </ul> </li> Display Dark theme on</li> no schedule - (let's give this a try, we'll see what it's like in broad daylight)</li> pure black - on (better for battery and I think it looks cooler)</li> </ul> </li> Night Light Schedule</li> sunset till sunrise - this is an experiment, we'll see if it is sensible at it</li> </ul> </li> Advanced (because this is extreem l33t haxxor stuff, beware n00bs) Screen timeout - 5 mins (scary stuff I know, you were warned by the advanced</strong> thing, I hope you know what you're doing. The nanny phone-state will keep you safe, don't you worry your pretty little head about this section. Twunts.)</li> Styles and wallpapers - yes you read that right folks, wallpapers are an "Advanced" feature. What is wrong with this people?</li> push a couple of images across to ~/Pictures with syncthing and set them as home & lock screen</li> Lock screen</li> Add text on lock screen Owner Tim Abell <tim@timwise.co.uk></code> (the idea is that people are generally good and this might be the best way to reach me if I've lost it)</li> </ul> </li> Tap to sleep - off (I only ever do this by mistake)</li> Font size > Largest (I'm gettingn quite long-sighted)</li> </ul> </li> </ul> </li> Sound</li> Vibrate for calls - Always (also has a vibrate then ring which is a nice option)</li> Advanced (why??????) Phone ringtone - thriller three (there's a great selection & variety available, impressed with Lineage here. Also liked Solarium, Sheep (yes actual sheep), Rigel)</li> Default notification sound - Hojus (also liked Beryllium and Titan for minimal stress-inducing feel. Again a great selection available with some really fun sounds).</li> Default alarm sound - Piezo Alarm (for the retro kicks. Again more winners in here, well worth exploring)</li> Dial pad tones - off</li> Screen locking sound - off</li> Touch sounds - off</li> </ul> </li> Privacy</li> Show passwords, display characters briefly as you type - off (I just find this annoying/distracting)</li> Notifications on lock screen Don't show notifications at all - this is more for a clear mind rather than privacy, though that's a bonus. I hate looking at my (locked) phone for the time and getting drawn into the notifications. Happened toooooo many times.</li> </ul> </li> Trust SMS message limit - always confirm (default of 30 seems like a lot of spam to me), might go to 5 if this is a pain</li> </ul> </li> Security</li> Screen lock - pin</li> Fingerprint</li> System Buttons Additional buttons Slider top - total silence</li> </ul> </li> </ul> </li> Status bar Network traffic monitor</li> Display mode > upload and download</li> Battery status style - Circle (much higher fidelity information)</li> </ul> </li> Advanced Gestures</li> Power menu Advanced restart - on</li> </ul> </li> Developer options Automatic system updates > off ("Apply updates when device restarts)</li> </ul> </li> </ul> </li> </ul> </li> About phone</li> Emergency information (wtf is this in about phone for? I had to ddg to find it) Add information</li> set all the things</li> Add contact</li> doesn't work, just shows empty list to choose from. dang. TODO</li> </ul> </li> </ul> Legacy phone connectivity</h3> The old phone has Wifi at home but without a sim has no internet out and about. Tether to new phone to give it access to internet elsewhere as needed.</p> Connected to WiFi hotspot but although it accepted the details it fails to actually connect.</p> Pair to other phone with bluetooth (either end is fine). On old phone open bluetooth settings, go to device > configure (cog) > Internet access > On.</p> Success, old phone has internet via bluetooth to new phone just like laptop. Win.</p> Todo</h2> firmware update: https://wiki.lineageos.org/devices/lemonadep/fw_update</a></li> Backup/restore with titanium</li> better camera support? https://www.xda-developers.com/google-camera-port-hub/</a></li> see if android pay etc will work with magisk magic mask pretending we haven't unlocked anything</li> find an alternative to titanium backup https://forum.xda-developers.com/t/farewell-to-titanium-backup-looking-for-alternative.3932814/</a></li> contribute to lineage to link to golang extractor, maybe with step by step instructions</li> Make signal default sms app</li> Camera from lock screen</li> transfer paid version of rain alarm, seems to be missing three-dot-menu on new version</li> Xmeye cctv</li> Backup alarms</li> Signal SMS warnings -> Settings-privacy-Trust-SMS Sending Limit > 5</li> No access to work google cal</li> Ignore public wifi</li> Maps.me voice. Needs language pack. Wut.</li> Remember Bluetooth tethering setting. Needs code change.</li> Phone default to Bluetooth</li> </ul> Problems & bugs</h2> Phone app Fails to unblank when phone rings half the time. Workaround to use power button to sleep/wake a couple of times to wake it up. Painful but bearable.</li> </ul> </li> Rainalarm problems: no menu so can't enable paid copy</li> map loads but no rain shown</li> </ul> </li> </ul> OS Updates</h2> https://wiki.lineageos.org/devices/lemonadep/update</a></p> Watch out for de-rooting, make sure backups are up to date first.</p> Got the first Over The Air (OTA) (...well, over wifi anyway) update from LineageOS. Download. Install. Reboot. Magisk no longer installed. Well that's a fucking pain. NeoBackup fails to start. Backups now broken. Fuck you android.</p> todo: see if I can restore root with out a full wipe and reinstall this time. twunts. what a pile of yaks.</p> OnePlus Community forum: Keep Root with Magisk with OTA updates</a></li> piles of fucking yaks courtesy of reddit</a></li> https://android.stackexchange.com/questions/200410/how-do-i-update-an-ota-while-retaining-twrp-and-magisk-only-with-my-phone</a></li> https://topjohnwu.github.io/Magisk/ota.html</a> -</li> </ul> "When an OTA is available, first go to (Magisk app → Uninstall → Restore Images). Do not reboot or you will have Magisk uninstalled."</p> </blockquote> Bit fucking late now.</p> So basically you have to know all this before</em> you let android install an update and fuck up your setup (again). Pit of failure.</p> Guess I'll have to re-do the patch thing I did in the first place.</p> Export Antennapod state, backup andOTP and make sure syncthing has pushed the files before attempting anything.</p> Re-root attempt - success!</h3> Let's try repeating the magisk setup from the original install but with the updated image.</p> Check current version installed on phone: settings > about > android version > > lineageos version: lineage-18.1-20220518-microG-lemonadep</li> </ul> </li> Download matching image to laptop from https://download.lineage.microg.org/lemonadep/</a> (if it's still available, which it is. If it wasn't then might have to run more updates or grab image from phone somehow)</li> Run the payload extaction, patch on phone & reflash as in original install: ../../payload-dumper-go/payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go ../lineage-18.1-20220518-microG-lemonadep.zip</code></li> [plug phone in to laptop with usb]</li> adb push extracted_20220704_232617/boot.img /sdcard/Download/</code></li> patch the file in magisk on the phone (install > patch > select file etc.)</li> get the patched file back on to laptop: patched=``adb shell ls /sdcard/Download/magisk_patched`` && echo $patched && adb pull $patched</code></li> advanced reboot on phone > fastboot</li> flash the boot image:</li> </ul> </li> </ul> tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload2</span></span> $ fastboot flash boot magisk_patched-24300_nspcI.img</span></span> target reported max download size of 268435456 bytes</span></span> sending 'boot_b' (196608 KB)...</span></span> OKAY [ 5.675s]</span></span> writing 'boot_b'...</span></span> OKAY [ 0.644s]</span></span> finished. total time: 6.320s</span></span></code></pre> reboot phone ("reboot system now" > power button)</li> open up magisk app .... drumroll ....</li> success!! "Installed 24.3 (24300)"</li> </ul> ... run a backup with neobackup to see if it works... oh never mind it just popped a notification that neobackup has been granted superuser so I guess it can run its backups now and the schedule has kicked it off after a reboot.</p> OTA (Over The Air) update without losing root - fail</h3> Having regained root by patching and flashing boot.img again (above) I'm going to try the OTA on-phone patching as per https://topjohnwu.github.io/Magisk/ota.html</a>:</p> Settings > System > Advanced > Updater</li> </ul> Press refresh symbol in top right corner.</p> Two images available:</p> LineageOS 19.1, 11 June 2022, 1.3GB > INFO > "Update blocked This update cannot be installed using the updater app. Please read https://wiki.lineageos.org/devices/lemonadep/upgrade</a> for more information."</li> LineageOS 18.1, 18 May 2022, 1.1GB > INSTALL > ...</li> </ol> So let's try the point update following the magisk ota instructions</a></p> Magisk app > Unistall.... um, where's that then. Perhaps missing because it wasn't instaled this way. "This will restore partitions modified by Magisk back to stock from backups made at install in order to pass pre-OTA block verifications." - yeah that figures, I didn't do that so there's no backups to restore.</li> Guess we'll skip this and hope for the best</li> </ul> </li> Settings > System > Advanced > Updater > 18.1 18 May 2022 > INSTALL > OK</li> ... ~ 10 mins elapse ...</li> Updater shows "reboot" button. DON'T PRESS IT!!</em>.</li> Magisk app > install > install to inactive slot (after OTA) > "Attention. Your device will be FORCED to boot to the current inactive slot after a reboot! Only use this option after the OTA is done. Continue?" > OK</li> Target slot _a ...</li> failed! dammit.</li> press little save button top right to save log</li> </ul> </li> </ul> Failure log: magisk_install_log_2022-07-05T21_12_57.log</code></p> - Target slot: _a</span></span> - Target image: /dev/block/sde16</span></span> - Device platform: arm64-v8a</span></span> - Installing: 24.3 (24300)</span></span> - Unpacking boot image</span></span> Parsing boot image: [/dev/block/sde16]</span></span> HEADER_VER [3]</span></span> KERNEL_SZ [40823296]</span></span> RAMDISK_SZ [10201092]</span></span> OS_VERSION [11.0.0]</span></span> OS_PATCH_LEVEL [2022-05]</span></span> PAGESIZE [4096]</span></span> CMDLINE []</span></span> KERNEL_FMT [raw]</span></span> RAMDISK_FMT [gzip]</span></span> VBMETA</span></span> - Checking ramdisk status</span></span> Loading cpio: [ramdisk.cpio]</span></span> - Stock boot image detected</span></span> - Patching ramdisk</span></span> Loading cpio: [ramdisk.cpio]</span></span> Add entry [init] (0750)</span></span> Create directory [overlay.d] (0750)</span></span> Create directory [overlay.d/sbin] (0750)</span></span> Add entry [overlay.d/sbin/magisk32.xz] (0644)</span></span> Add entry [overlay.d/sbin/magisk64.xz] (0644)</span></span> Patch with flag KEEPVERITY=[true] KEEPFORCEENCRYPT=[true]</span></span> Loading cpio: [ramdisk.cpio.orig]</span></span> Backup mismatch entry: [init] -> [.backup/init]</span></span> Record new entry: [overlay.d] -> [.backup/.rmlist]</span></span> Record new entry: [overlay.d/sbin] -> [.backup/.rmlist]</span></span> Record new entry: [overlay.d/sbin/magisk32.xz] -> [.backup/.rmlist]</span></span> Record new entry: [overlay.d/sbin/magisk64.xz] -> [.backup/.rmlist]</span></span> Create directory [.backup] (0000)</span></span> Add entry [.backup/.magisk] (0000)</span></span> Dump cpio: [ramdisk.cpio]</span></span> - Repacking boot image</span></span> Parsing boot image: [/dev/block/sde16]</span></span> HEADER_VER [3]</span></span> KERNEL_SZ [40823296]</span></span> RAMDISK_SZ [10201092]</span></span> OS_VERSION [11.0.0]</span></span> OS_PATCH_LEVEL [2022-05]</span></span> PAGESIZE [4096]</span></span> CMDLINE []</span></span> KERNEL_FMT [raw]</span></span> RAMDISK_FMT [gzip]</span></span> VBMETA</span></span> Repack to boot image: [new-boot.img]</span></span> HEADER_VER [3]</span></span> KERNEL_SZ [40823296]</span></span> RAMDISK_SZ [10715780]</span></span> OS_VERSION [11.0.0]</span></span> OS_PATCH_LEVEL [2022-05]</span></span> PAGESIZE [4096]</span></span> CMDLINE []</span></span> - Flashing new boot image</span></span> - Sepolicy rules dir is /dev/hWU/.magisk/mirror/metadata</span></span> ! Installation failed</span></span> ! Unable to download bootctl</span></span></code></pre> Guess I'll try the patch of boot again.</p> Conclusion: inconclusive</h2> The main thing I've learned from this is the long-standing duopoly of iOS+Android has caused deep and hard to reverse problems in the phone software ecosystem. It's a crying shame really because there is so much opportunity for innovation now that phone hardware is basically done, but instead we get stagnation, pointless features, anti-features and down-right user-hostile behaviour from both we-know-best camps. It really reminds me of the dark years of the browser and operating system wars. Particularly when internet explorer became dominant and website (i.e. app) developers targeted proprietary IE APIs, locking everyone in and nearly killing the competition. The same for windows in its prime (when Balmer shouted developers-developers-developers he knew the apps created platform-vendor lock-in that he so desired).</p> There are some promising signs: Ubuntu so nearly made a phone, and then didn't hit their funding target (the ubuntu phone operating system lives on in obscurity), The Librem looks like it might be a sustainable Linux based phone & business (Android is not</em> Linux remember).</p> For me I think I'll do the closest thing I can to dual-boot and VMs which is to have a second burner phone to run prioprietary crap, but have a primary phone that's as open as I can make it, just like I did with Linux and Windows around 2005-2010.</p> Every person that takes this route is a vote for change. User numbers matter, especially when it comes to app vendors; they won't invest in dead platforms, but they can't and don't ignore growing market share, and if that's free and open it gets supported. Just look at Microsoft rewriting their entire</em> dotnet ecosystem from scratch to be cross platform.</p> Open source moves slowly, but like an iceberg it's hard to stop when it's claimed the land.</p> End... for now</h2> That's as far as I've got so far. I'll be editing this post as I progress with the install so do come and look again. Also suggestions and questions welcome.</p> I really do hope one day I can de-google my phone just like I de-microsofted my laptop, but today is not that day for me.</p> Further reading</h2> https://www.howtogeek.com/358166/using-android-without-google-a-kind-of-guide/</a></li> </ul> Being a development team lead 2021-12-10T00:00:00+00:00 Here's my take on what it means to be a dev-lead / tech-lead / lead developer; and why it's time I stepped up.</p> I'm going to explain what I think makes for a really good high-functioning dev team, and how to get there along with links to essential reading on the subject.</p> As always contributions and thoughts welcome, drop me a line. If you want me to help with your team just shout. I love to just chat with people about their challenges in digital delivery, whether or not I end up becoming part of the project.</p> I'm also going to say why I think it's time I shifted my</em> contracting focus from individual contributor to being a lead dev for teams.</p> How to be an excellent lead dev</h2> Having worked under many team leaders of all types I have a pretty good idea of what's pleasant and effective. And I'm glad to say that being a nice human being to be around is a big part of being a great leader. Dictatorial, unpleasant and command-and-control leadership are just not the right tool for running complex digital delivery projects</p> Servant-leader</h3> The best leaders I've come across are almost deceptively non-leader-like. This is because our culture in the west (hello press, television and film) portrays "leaders" as hard-ass bullies who get the best out of an organisation by ordering people around, knowing all the answers like some kind of god and generally being unpleasant. The reality is that the best leaders don't lead by intimidation, they lead by example, lead by being humble, lead by coaching (see below) and know that relationships and trust are what really matter for teams that can ship and respond to problems.</p> Culture of trust</h3> Everyone in the team is doing the best they can with the knowledge and circumstances they have. We can all help each other get better with trust and a good dose of nonviolent communication</a>.</p> Openness with all</h3> Make things open: it makes things better</a>.</p> I've always thought it best that anyone in the organisation who is interested can see every last detail of what we're working on, right down to the code. This does of course require a culture of trust, but if that's missing there are deeper problems to be solved.</p> There are of course some things that need to be protected such as production keys and the personal data of users.</p> As an example I ran an end-of-sprint demo for a client with all the major stakeholders present for an important project. The demo consisted of a "red screen of death", which showed that we had been unable to get a particular piece of technology with which the team was unfamiliar to work even after a whole sprint on it. Once the shock of the blunt honesty subsided the senior stakeholders were able to see that something needed to be done to ensure success and helped us recruit an external consultant with expertise in that particular technology. From there on things went much more smoothly. If we hadn't demoed our "failure" then there was a real risk that the senior stakeholders would have mistakenly thinking everything was fine and now action was needed, which would likely have resulted in a complete failure of the project to deliver. Some people seem to like to always paint a positive picture to their seniors (the RAG reports that are always yellow until it's too late to fix anything and they finally all go red, as they people go and get new jobs elsewhere...).</p> Setting the standards</h3> Nobody wants bugs, unmaintainable spaghetti code, regressions due to missing test coverage. But how do you achieve this in a collaborative positive way? Just demanding it be so is ineffective and builds resentment.</p> No team is perfect. No software is perfect. There is only "better", "worse" and different tradeoffs between different pressures on system design and human behaviour. So we should strive for the best, bring the team with us, and accept that we are all flawed human beings with an inability to simultaneously keep in our heads every constraint and piece of the system. In fact software is largely written and architected the way it is because</em> even the best human brains have limited capacity for knowledge of a system. Managing complexity and coping gracefully with humans being human is an important part of the puzzle.</p> I see the push towards today and tomorrow's "best practice" more as giving permission to the team to strive for it than enforcing. Most developers I know want to be and do the absolute best they can. When team output doesn't meet some criteria (such as test coverage) these are some possible reasons that I'd look at. I'd use the ordering below so that we can assume good faith until proven wrong. I definitely wouldn't start with confrontation or accusation. Assuming the best does not mean giving up powers of last resort should all other options be exhausted, but that would be the last option. Filtering for great people on the way in is much better than undoing poor selection once the team is up and running.</p> What pressures are being placed on a contributor to behave a particular way, where is that coming from? E.g. is a delivery manager pushing for raw speed and that being interpreted as a need to skip test coverage and just ship bare production code. In that case perhaps some conversations about the tradeoffs being invisibly made are all that it would take, and some defense of the fundamentals that drive the need for "best practice" in the first place (e.g. regression-proofing to allow continuous delivery, and confidence in refactoring to support agility and ongoing quality, see also XP</a>)</li> Is there a lack of knowledge as to why this good practice is beneficial to the project? If so some mentoring and coaching might be in order. Maybe as part of some pair programming exercises.</li> Provide formal training (perhaps Pluralsight courses) to fill in the knowledge gaps. Particularly for permanent employees. For contractors perhaps a suggestion of courses they could take to level-up.</li> Communicating to those involved in line management / supplier management the concerns with not meeting the standards the rest of the team are able to achieve. Discussing additional ways of providing support to enable the individual to level-up.</li> Finally and this is not a preferred option due to the disruption it causes make the call to formally put someone on a formal improvement plan or move them on.</li> </ol> The coaching ladder</h4> Inspired by Developing Leadership podcast episode 19 about a strong engineering leadership culture</a>.</p> Different people, cultures and neurological wiring need different ways of setting out expectations. A two-pronged approach of gentle coaching + clear expectation & consequence setting seems to cover all bases.</p> Listening mode - make sure people are heard in groups and one-on-one</li> Gently coach expectations - "have you thought about doing x?" / "I observe that your PR descriptions average 10 words" / "here's how writing clear PR descriptions helps your valuable ideas be recognised by the team"</li> Set out consequences of not meeting expectations - "We require excellent communication in pull requests to the standard of [example]" / "this standard is a requirement of continuing to be a senior engineer here"</li> </ol> One-minute manager</h4> Where course correction is needed (when operating more as a line-manager) The (new) One Minute Manager</a> offers some great advice on keeping it simple and clear for all involved. This suggests that one-to-one interactions take the form of:</p> One-minute on goals</li> One-minute on praise</li> One-minute on redirection</li> </ol> If doing only three minutes of this means they can be done more often and more reliably with the whole team this is a win, as often the opportunity for this kind of discussion is lacking due to the pressure to ship. It's important to note that this doesn't mean the relationship has reverted to command and control, only that clear expectations are important for everyone's success and happiness.</p> Current "Best Practice" and Lofty Goals</h3> Things we should strive towards doing/having.</p> Great test coverage 100% unit test coverage (in meaning not just in line count) Why? Because this enables refactoring with confidence, reduces the chance of regressions and encourages good design (e.g. decoupling, SOLID etc.)</li> </ul> </li> Use TDD (Test-Driven Development)... if it's the right tool for the job. TDD is a useful technique to be sure. It's harder to use when you can't picture the shape of what you are going to create in your head in advance which I think is probably its biggest weakness. The ultimate objective is that it should be impossible to change the intended behaviour of a production code without a test failing. TDD helps with this but isn't the only way. Another way is to alter the behaviour temporarily (locally) after the code is written and make sure the tests do actually fail. So long as the output coverage is good then I don't personally think that being religious about TDD is especially helpful. I think it's a technique that all developers should learn and practice, whether or not they use it for every piece of code they write.</li> Smoke tests Some kind of end-to-end "did we break anything critical" that runs on real infrastructure. This usually requires dedicated effort to create beyond the tests created during individual feature work. I recommend putting this in the backlog and ensuring it is scheduled as it's too big for a hero-developer to "just do" unless you've completely lost control.</li> Visual regression testing</a> - in anything with a UI this can catch regressions and unintentional changes that would be hard to catch.</li> </ul> </li> "BDD" (Behaviour Driven Development) aka specification by example</a></li> A tool for a job. Use it where it's useful such as allowing business analysts to create and validate example scenarios for complex calculations or flows. Beware of the overhead of example wiring code.</li> Often confused with UI tests due to coincidental timing of creation with browser-automation. BDD is not the same as UI testing, mkay?</li> The point of BDD is more that it's a user/business need focussed description of what the software is supposed to do that's then made executable through a translation layer. Do not forget this.</li> You can get a lot of the benefit of BDD by just writing test method names in the BDD style</a>, such as def the_order_total_should_be(total_price)</code> and then chaining them together in a top level "scenario" method just like you would with gherkin but without all the example parsing overhead.</li> </ul> </li> Continuous delivery (and confidence to ship any time without even looking) Lights-Out delivery</a> - i.e. never having to manually interact with production servers. Related to configuration-as-code etc.</li> Use feature flags (related to the true meaning of the oft misused phrase "continuous integration") to avoid long-lived feature branches for things you can't release to the users yet.</li> </ul> </li> Use the best type of project management (yes I used the dirty words!) for your</em> project. Agility: Methodologies are fine, but the point is to be able to be responsive to changing knowledge and business requirements. Never forget that.</li> eXtreme Programming</a> - an important piece of the landscape, with a focus on technical approaches that allow agility (always shippable, fully tested, etc.)</li> Scrum</a> - a good basis for all projects, especially greenfield development</li> Kanban - better for maintenance mode projects It's very important to understand "Work In Progress (WIP) limits" regardless of whether you use kanban or not. This video explains wip limits</a> in a way that is easily understandable. The mathematically truth of wip limits will always be counter-intuitive to my monkey-brain I think. In short, throughput matters more than utilisation of each person/resource.</li> </ul> </li> Lean</li> Flow</a></li> Scaled Agile (SAFe)</a> for multi-team agile</li> ShapeUp from basecamp is inspirational. It's a refreshing change from scrum. I wrote a summary of ShapeUp</a> which, in my humble opinion, is worth a read.</li> Waterfall is bad. Period. Gantt charts are to be treated with suspicion, along with anyone and anything else that promises complete predictability of software development.</li> Self-directed teams that own their results and can use appropriate methods for their particular challenges are far more effective than standardization.</li> Personally I like some kind of blend of scrum and kanban, and definitely no story pointing.</li> </ul> </li> </ul> </li> Lightweight agile governance: Stakeholders showing up to sprint demos (show-n-tell) is much more time-efficient and more powerful than written reports.</li> </ul> </li> Ability to rebuild entire estate from source-control: Binaries</li> Infrastructure (configuration-as-code)</li> </ul> </li> DevOps in practice not just in name (the development team owning</strong> production) This implies a diversity of skills within the "development" team.</li> Assistance from people with expertise outside the team is of course welcome, but the important thing is the team continues to own their own outcomes.</li> </ul> </li> Some (but not complete) harmonization of technology choices: Largely consistent technology makes hiring and moving around projects easier</li> Completely consistent technology prevents using the best tool for the job</li> </ul> </li> Following 12 factor principles</a>. Pushing beyond the original 12 factor principles - Small Batches podcast episode 7</a></li> Continuous improvement (of the team, the processes and the system being built)</li> </ul> </li> Pair/mob programming. Collaborating on just about everything ensures no-one is a single point of knowledge for any one thing.</li> The sharing of knowledge, skills and techniques improves everyone's skills as we go.</li> Many problems and bugs are caught even before git add</code> has been run. The later a problem is found the more it costs to rectify it.</li> </ul> </li> </ul> Estimates</h3> Story points, t-shirt sizes, predictions, time-based estimates and no-estimates.</em></p> This is a topic that always causes heated debate with lots of dogma on all sides. Experiences of day-long point-guessing sessions leave developers who just want to ship nervous of anyone who mentions the word estimate or point. Project owners/managers get nervous when they have no idea how much they are about to spend. Deadlines can be real or made up. Cost/benefit decisions have to be made. So what should we do?</p> I say it depends. And more usefully, only create estimates if you can prove you need them</strong>, and then use the most lightweight way you can come up with of getting just enough information to make a call and go back to cutting code.</p> Creating estimates has a cost, make sure you know you need it otherwise don't pay the price. Kanban calls this "waste</a>" (i.e. anything that doesn't end up in the final product, which definitely includes estimation work).</p> Avoid story points</a> - they were basically invented as a way to prevent people mistaking estimates</strong> for commitments</strong> but have taken on a horrifying life of their own. I have never</em> seen them be particularly useful in any of the many projects I've been on, and I have regularly seen them be a big cause of pain, wasted time and discontentment.</p> Genuine reasons for needing some kind of estimate:</p> Big-feature prioritization - use t-shirt sizes (small, medium, large, elephant)</li> Can't miss deadline (I arrived on a project when the release date of a non-existent thing had already been announced by the managing director to all the customers). Do as few of the following as you can: (listed in increasing order of cost and decreasing order of preference.) A "finger in the air" hunch for viability of the project from an experienced engineer</li> Consider just doing the minimum viable product (MVP) first and then adding nice to haves once that's done (iterate). Focus on always-shippable.</li> If you are concerned even the smallest viable thing might not be doable then you might have to create some architecture designs and prototypes up-front to reduce risk of discovering complications later.</li> If time is tight and even the MVP is looking uncertain then you might have to create actual estimates for more detailed tasks to give you a better idea of the options ahead of you. I have actually done this on a fixed-deadline project. It's time consuming and dangerously waterfall but it might be your only choice. The more granular the task list the more accurate the estimate (it helps compensate for human bias), but on the flip side the more detail you create without building the more likely you are to be wrong about the design and tasks and for it all to be a waste.</li> </ul> </li> </ul> Planning releases - do you really</em> need to do this give the cost? Consider using feature-flags instead to hold back features if you need to time their release.</p> If you need to forecast based on points use monte-carlo story-point forecasting as implemented by Chris Bimson</a>. While I like this approach I'd challenge the need for the forecasts in the first place before committing to the significant extra overhead of calculating and tracking points and then turning that into forecast.</p> Staying sharp</h3> Even though I accidentally led a couple of teams early in my contracting career I didn't at that point fancy moving "up" the ladder. I had seen people become "dev team managers" and rapidly lose all their technical skills through a combination of atrophy and the never-ending churn in state-of-the-art technology and practice. I certainly didn't want to end up us a mediocre manager with mediocre and outdated tech skills. So I put my head down and kept on coding as an "individual contributor" (sometimes referred to as an I.C.). Since then I've learned and seen some tricks that I think can allow people to stay sharp no matter how high they climb.</p> The pendulum</h4> By alternating roles you take on between individual contributor (coder) and team lead roles you can keep the experience of coal-face high-pace shipping fresh whilst also giving more value by helping whole teams succeed.</p> https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/</a></p> The non-critical-path task</h4> I tried picking up a story from the sprint when I was leading a team once. It didn't go well but I didn't know any better. Because my primary goal was team enablement and my time was interrupt-driven I'd end up with the team waiting for my piece to get done. Not a great help to the team's delivery speed. I also found with the team being full time on code and me less than full time I was always just that little bit behind on current thinking in the heart of the codebase, and you don't have to be much out of step to be much less productive or cause more problems than you solve.</p> Since then I've seen two great approaches implemented by others:</p> Take on a non-critical but useful coding piece that won't block the team. I watched an amazing product owner (who can also code) built an entire support console microservice while us devs built the main production app. This gave huge benefit of allowing the support folks and BAs to self-serve on things we didn't have time to build to production grade. It kept him close to the actual product. It also kept his Ruby on Rails skills current and sharp. Ace. I shall be copying this trick!</li> Work on enablement pieces. Make sure CI pipelines are flowing well. Do version upgrades. Tackle bugs that haven't made the team's sprint. I've seen a team lead do this effectively and would recommend.</li> </ol> Interviewing / filtering</h3> Not everyone gets this luxury but being allowed to control who works in your team will do more for your ability to succeed than anything else.</p> For a phone screen I wrote up my process here: Technical phone screen interview questions</a>.</p> Beyond a quick phone/video screen my favourite approach is a pair-programming exercise as this is such a rich source of information and matches how we actually get things done.</p> Coaching & mentoring</h3> I can't say it better than The Coaching Habit by Michael Bungay Stanier</a>. The follow-up book The Advice Trap</a> is also excellent.</p> I've often been guilty of unloading all my "knowledge" on others before they've even asked. Now I see the error of my ways and instead as per the book look to nudge and help at people's own pace. Being there to assist and give confidence as needed; leading people to the best answer through questions instead of answers (and being happy to be surprised that it's a better answer than I had in mind).</p> Decision making</h3> It's good to have single points of responsibility such as a tech lead or project owner, however that doesn't mean that it's top down control.</p> As for how to make the best decisions in chaos and complexity, or when thinking seems to have narrowed to group-think this is an excellent guide: Decisive: How to Make Better Choices in Life and Work by Chip & Dan Heath</a></p> Things to read / watch / listen too</h3> Blog posts Maker's Schedule, Manager's Schedule by Paul Graham</a></li> Being Glue post/talk by Tanya Reilly</a></li> Command and Conquer and the Herd of Coconuts by Joel Spolsky</a></li> Normalization of deviance by Dan Luu</a></li> </ul> </li> Tech leadership books Notes to a Software Team Leader: Growing Self Organizing Teams by Roy Osherove</a></li> </ul> </li> General leadership & communication books The Advice Trap: Be Humble, Stay Curious & Change the Way You Lead Forever by Michael Bungay Stanier</a></li> 5 Dysfunctions of a Team</a></li> The First 90 Days</a></li> </ul> </li> Personal development (you'll need this for you and your team!) Daring Greatly by Brené Brown</a></li> Nonviolent Communication; Marshall Rosenberg</a></li> Bonds That Make Us Free; C. Terry Warner</a></li> 48 Laws of Power; Robert Greene</a></li> </ul> </li> Lean / Flow / Kanban agile etc The Goal: A Process of Ongoing Improvement Kindle Edition by Eliyahu M. Goldratt</a> (Audible version has sound effects and everything!)</li> WIP: why limiting work in progress makes sense (Kanban) (YouTube)</a></li> </ul> </li> </ul> (Personally I like Audible books, I think it sinks in better being told the stories by another human voice)</p> Who to learn from</h3> The legend of Joel Spolsky</a> - I've been reading his articles since I first started coding. Remember "the joel test</a>"?</li> </ul> Other ways of moving "up"</h2> These days thankfully moving into management isn't the only way to get a pay raise in many organisations. Places have recognised that getting ever deeper into technology is in itself valuable, spawning titles such as "principle engineer" that carry the weight and renumeration to match the value and experience of a coder who has seen many things and developed many other skills beyond pure coding.</p> There's also the "architect" route where you can be an expert in deciding what to use and how to put it together from the seemingly infinite choice of technical solutions, patterns, services and infrastructure that will solve pretty much any given problem. There's good money to be made being an architect, if nothing else than because your power to royally screw everything up for everyone that comes after you is probably about as high as it gets in this role! (Yes I've been on the receiving end of good and bad "architecture").</p> The road to CTO. You can join a small startup and ride the explosive growth, ending up with many engineers and services under your control. Or you can work your way up relentlessly in one or more large organisations.</p> Me, a leader?</h2> Why me, why now, why this?</h3> I'd like to build a product, but I can't just stop contracting and hope for the best. As such I've spent some time considering which direction I should take my contractor life.</p> When considering changes of career direction I highly recommend the book "Working Identity" by Herminia Ibarra</a> which lays out a framework for thinking about the journey of the change, combined with lots of research and stories of others who've faced the chasm (often but not exclusively in the traditional mid-life point).</p> Here's some ideas that popped up ("identities to try on" in the terminology of the book):</p> Systems-integration agency/consultancy</strong> - I had a go at this but seems to be too big a jump for me right now, on the plus side it's kicked off another period of learning and self-development for me and is the reason I hired a business coach</a> which I am eternally grateful for.</li> Become a "frontend" engineer</strong> - Traditionally I've been a "full stack" or "backend" engineer which fits my engineering strengths of dealing with complexity in code (for example tech debt as discussed here</a>). Way back there wasn't much complexity beyond a bit of jQuery in most frontend things, in fact I didn't come across the frontend/backend specialization for many years. For a long time the designers-who-can-html dominated frontend land. But these days with ES6, Typescript, npm, angular, react, etc. etc. there is now endless complexity on the frontend for someone like me to get my teeth into. With that in mind (and with a view to improving my ability to ship side-projects) I've been updating my languishing frontend skills. (There's not much frontend in my recent work for UK Gov as that's pretty much templated out with the gov.uk design system</a>, and accessibility concerns means that fancy responsive SPAs don't tend to pass muster). Having put the idea out there with recruiters etc. but not getting much interest I think it's probably a jump too far in one go, but either way I'll spend some spare time skilling up.</li> Lead developer</strong> - I don't really know why I didn't think of this sooner, it's actually a good fit for where I've been and where my life is at now. This is my current identity to try on, wish me luck! The rest of this post is about this.</li> </ul> Here's why I like the team lead idea right now:</p> I think can provide more value as a contract team-lead for dotnet delivery teams. I can still crank code, and still enjoy coding, but helping a whole team succeed is a worthy thing.</p> The higher rates would make up for it being a less commonly available role (there's always at least two individual contributors for every team lead, and leads are less commonly contractors). And I could always fill in with coder work in any gaps.</p> Thinking back to my history and current position, I've enjoyed the lead roles I've done, I love enabling those around me, and I've learned some tricks for being able to stay close to the tech without getting in the way (see "staying sharp" above). I can still crank code, but it seems a shame to learn everything I have beyond the code and then not put it into practice.</p> Experience leading</h3> One of my earliest contracts was a project where I was brought in as a solo dev with a view to delivering an asp.net project. The first task was to collect the requirements from a sales person (not an end user or user researcher sadly but that's how it was). So armed with my new love of balsamiq mockups</a> I set about turning the explained needs into a set of wireframes, using them as a tool for a tool for discussion and iterating them as I better understood the problem at hand. I then started work building. After some time the CTO sauntered over and casually asked how likely we were to meet the publicised launch date now that we understood what needed to be built. "Zero" I said, as this was quite a sizeable thing on a not particularly long timeline. A lovely chap to work with he didn't pass further comment and disappeared again.</p> Not much later I found myself with a team to work on the project consisting of a mix of permanent development staff who'd been reassigned to the project. Because I was there first and I was in touch with the people relaying user's needs I was naturally the hub for the team. Additionally I was one of the few who'd practiced agile methods in previous roles. The result was I felt it was far more important that I enabled the rest of my team to be as productive as possible by eliminating things that were blocking them as my top priority. So there I was, being a team lead for a dev team building a new product, making sure everyone was able to get on, talking to stakeholders and running our sprint demos to ensure everyone inside and outside the project team understood how it was going and could play their part. Pleasingly we shipped a usable system in time for the deadline.</p> Experience mentoring</h3> I've always helped everyone I can be the best they can, freely sharing knowledge with anyone who's interested. Not just developers either, in my view the more non-technical roles know about the thing we're all working on the more effective they can be.</p> I officially mentored DfE Digital's first civil-servant developer which was a real pleasure. You can hear a bit more about that on my podcast episode "A retrospective of mentoring, with David Gisbey"</a></p> Summary</h2> Well that was a lot more than I expected when I sat down to write, guess there's a lot floating around my head after 21 years thinking, doing and learning.</p> Wish me luck, and I hope this helps you in some way.</p> If you want my help with a team just get in touch, I'm always happy to chat.</p> Technical phone screen interview questions 2021-12-10T00:00:00+00:00 I was tasked with giving an initial assessment of technical competence / fit in a 30 minute phone call.</p> This tech screen is based on Steve Yegge’s "The Five Essential Phone-Screen Questions"</a> My lightweight version goes something like this:</p> 1) Coding</h2> Ask them to describe the code for a console app that prints the ten-times table in a grid. This is intended to get them to describe a nested for loop, or maybe some fancier construct. It doesn’t matter if they nail it, you can tell if they understand how to code from how they talk about it. Bonus for whether they can understand a verbal description (possibly imperfectly described), and throw together a solution in their head.</p> This isn’t a substitute for an in-person pairing session, but it gives a quick indication whether they have a clue how to code.</p> It's supposed to be something like this, which is remarkably hard to explain on the phone from my side (no points deducted for my inability to be clear!):</p> 1 2 3 4 5 6 7 8 9 10</span></span> 2 4 6 8 10 12 14 16 18 20</span></span> 3 6 9 12 15 18 21 24 27 30</span></span> 4 8 12 16 20 24 28 32 36 40</span></span> 5 10 15 20 25 30 35 40 45 50</span></span> 6 12 18 24 30 36 42 48 54 60</span></span> 7 14 21 28 35 42 49 56 63 70</span></span> 8 16 24 32 40 48 56 64 72 80</span></span> 9 18 27 36 45 54 63 72 81 90</span></span> 10 20 30 40 50 60 70 80 90 100</span></span> </span></code></pre> (gist</a>)</p> 2) OO design (and other structures)</h2> Ask them to describe how they can arrange their code, giving a hint that I’m after object oriented things. I’m looking for a confident off-hand description of inheritance, interfaces etc.</p> If they don’t fluff that, ask if they know non-OO approaches (composition, modules, mixins etc) that are used in languages other than C# to see how broad and deep their knowledge is.</p> 3) Scripting & Regex</h2> Context: It’s not uncommon for microsoft devs to never look outside the microsoft sandbox. This can result in someone spending week(s) writing a program to solve a problem that can be done with one linux command. Times are changing and Microsoft loves linux, but these narrow thinkers are best avoided. This question is almost pointless for ruby devs who live in unix land.</p> Question: you have a server with thousands of plain-text log files scattered across hundreds of folders and you need to find all the lines with a particular user-id in them to solve a production problem.</p> Good answer: grep</code> (this means they know of unix tools, maybe probe how wide that is) Bad answer: I’d start a new project in visual studio and use FileInfo classes…</p> Follow up question: what if you needed to find an identifier that matched a pattern but wasn’t always the same? (i.e. do they know about regex)</p> Great answer: I’d use egrep</code> with a regex (or similar tools); goes on to explain a bunch of great tools I hadn’t heard of before</p> If they have other smart ways (e.g. they know about log aggregation tools like Kibana), guide them back to a limited toolset (“it’s a legacy system, you haven’t got any of that...”)</p> 4) Data Structures</h2> Do they know the difference between dictionaries and lists, particularly when it comes to speed of lookup.</p> Related: Do they know about algorithmic complexity</a> - this sorts the self-taughts from the computer-science degree crowd, but if you don’t know this stuff it’ll bite you sooner or later.</p> See if they are aware of the available structures. See how clearly they can explain why you’d choose one over the other for a particular piece of code. It doesn’t matter if they can recall the specifics, it just matters that they’d think to go and look up the specifics when they needed to decide.</p> 5) Bits and bytes</h2> Do they know why you don’t store money in floating point variables?</p> Great: use decimal instead because floating point cannot accurately represent decimal fractions and will give you rounding errors that upset the accountants and customers Okay: always use decimals, but don’t really know why Bad: I’d use a float for money</p> So that’s it for Steve’s list.</p> I also like to ask them how they feel about the following:</p> 6) Testing</h2> Are they militant about TDD? What do they think about BDD? Are they more pragmatic?</p> How do they think about why you write tests.</p> This isn’t likely to be make or break, but might indicate better or worse fit with a team and is good to know.</p> 7) Agile</h2> What kind of agile structures have they worked in. What did they think about it.</p> I’m looking for enthusiasm for the way the best agile teams operate, it doesn’t matter too much if it’s SCRUM or something else. Do they seem like they’d be adaptable? Would they push for better?</p> Have they worked on GDS style teams? (For gov work)</p> Assessment</h2> During the debrief I usually discuss at least some of the following areas, focussing on what seems to be the make or break points:</p> skill / knowledge level: architect & team lead > senior engineer > narrow senior engineer > competent > hand-holding needed > complete liability</li> communication style, ability to discuss piece of theoretical code</li> attitude towards tech (e.g. do they demand testing or are they more “pragmatic”)</li> attitude towards and organisational structures and approaches (i.e. scrum or GDS team structures)</li> how they would likely fit in with the intended team</li> </ul> Some example outcomes of the technical assessment might be:</p> Strong - knew everything, broad knowledge, could clearly do the job</li> Qualified positive - can probably build something within some boundaries, but you’d want a strong tech lead in charge to keep things on track</li> Weak - lacking knowledge of important areas of tech, failed to explain a good solution to a simple nested loop problem</li> </ul> Bias and scoring</h2> To minimize risk of bias sneaking in and giving me a "favourite" based on how similar they are to me rather than how well they did I like to do a written report scoring each area above.</p> Integrating ShipStation with QuickBooks Desktop 2021-09-27T00:00:00+00:00 A possible customer of my new eCommerce Integration business</a> popped up with an interesting problem: how to have orders that are entered into QuickBooks Desktop</a> automatically create matching entries in ShipStation</a>.</p> "How can I get my orders from QuickBooks into ShipStation without my staff manually retyping them?"</p> </blockquote> What is ShipStation?</h2> It is a one-stop order shipping interface for shipping your physical outbound orders that can in turn talk to all the actual couriers so you don't need to deal with them individually. It connects to Royal Mail, Hermes, DPD etc.</p> Connecting ShipStation to QuickBooks Desktop</h2> There isn't a direct way of connecting them together, so that means we need something to do the work of pulling orders out of QuickBooks and pushing them into ShipStation.</p> Here's two pieces of software that can connect the two systems together:</p> Connex</h2> https://www.shipstation.com/partners/connex-quickbooks/</a></li> https://www.syncwithconnex.com/</a></li> Connex is a bit like Codeless BPA in that it enables a flow of orders, but specific to QuickBooks rather than having many connectors on both sides.</li> This page lists exactly what pieces of data will be transferred: https://help.syncwithconnex.com/hc/getting-started-with-shipstation</a></li> ShipStation help page on Connex for QuickBooks</a></li> The base price for Connex is $199/month</a> at time of writing.</li> </ul> Codeless BPA</h2> https://www.codelessplatforms.com/business-process-automation-platform/</a></li> Codeless has a ShipStation connector</a></li> BPA Platform starts from £175/month billed annually.</li> We'd need to do some investigation of the method of integrating QuickBooks Desktop with Codeless BPA.</li> </ul> Connex versus Codeless</h2> Both Codeless BPA and Connex have similar monthly costs, and both will need setting up, so choosing which to go with is going to come down to which option fits your businesses needs better now and in the longer term. Connex will solve the immediate problem, but Codeless has more connections you can use in the future.</p> DBSync</h2> https://www.mydbsync.com/workflow/shipstation-quickbooks-integration</a></li> Cloud and On-Premise offerings.</li> "Chat for pricing"</li> </ul> Unify by Webgility</h2> ShipStation integration: https://www.webgility.com/integrations/shipstation</a></li> QuickBooks desktop integration: https://www.webgility.com/integrations/quickbooks</a></li> From $39/month billed annually</a> at time of writing.</li> ShipStation integration help page</a>.</li> </ul> Automate.io</h2> https://automate.io/integration/quickbooks/shipstation</a></li> From free to $159/month</a></li> </ul> TradeGecko (retired)</h2> This is being retired</a>, but mentioned for completeness.</li> https://support.tradegecko.com/hc/en-us/articles/201825933-ShipStation-Integration-Setup-Guide</a></li> </ul> To integrate now, or to change technology?</h2> As everything moves to the cloud it's worth considering whether to invest further in your existing desktop-based technology choices, or to take the opportunity to move to cloud based accounting and ERP systems before looking to connect them together.</p> Get in touch</h2> Did I miss an option? Let me know!</p> Can I help you or someone you know with something? Reach out!</p> The death of custom software development 2021-08-18T00:00:00+00:00 I've been around this industry for a long time and have been quietly observing the bigger trends while I fiddle around with the details of software. Here's what I've noticed: writing software by hand is doomed. Expect that it isn't, and it's a bit more nuanced than that.</p> I think the industrial revolution and the creation and growth of engineering as a profession is a good way to think about the progression of the software profession in the large.</p> The long view - the birth of payroll software</h2> A few years ago I decided to get a bit stronger (being your typically feeble software engineer at the time) and joined a Nuffield gym and took them up on their personal training options so that I wouldn't injure myself picking up the first 1kg weight I found in the gym. The PT I got happened to be a lovely older chap named Bryan, who is a very calm and experienced trainer who really helped me build some useful real-world strength through the medium of increasingly heavy kettle-bells and other such devices, all without injury. Now obviously we got talking as we spent many hours on turning me into a proper human, and it turns out that Bryan used to work for the might HP (the computer company not the sauce) of all things, and was part of the software world for many years before I joined it amidst the dot-com mania. I have heard talk of people writing custom payroll systems in the past, but Bryan actually lived it and it was great hearing the stories. In many ways nothing much has changed; software engineers write custom software to solve some pressing business problem for the big businesses that can afford it, and everyone else gets by with bits of paper or whatever they can pull together. The software engineers complain about the hardware engineers, the hardware engineers complain about the software engineers, the sales people sell things that are expensive to do and the software engineers grumble and try and make good. There are fascinating stories of obscure bugs where you have to drill all the way down to the compiler or hardware to find the true source; the stuff of coder legends. All this was true then and is true now.</p> "That's all very well", you say, but what's that got to do with me? Well I think some of the systems Bryan worked on are illustrative of the geological shifts in software over the years, and are indicative of changes happening to this day and still to come.</p> In that era, there was no good off-the-shelf payroll software that could solve every business's payroll needs, and every business needs to pay its staff. Doing it by hand is laborious and error prone which makes it a good candidate for turning into software. The in-house custom coding crowd got there first and made it work for big businesses that could afford development teams or the equivalent consultancy fees. Then vendors started to show up but there were edges that they couldn't handle so there was still room for custom code, and now it would be [almost] madness to start a payroll software company (though you might find a way to disrupt the existing players like Sage, but it's not going to be all green fields for the taking like it was back then).</p> The trend of commoditization and the shrinking need for custom code</h2> "So what", you say, "there's still vast plains of untouched lands for software development, coding will never die! Coders unite!"</p> As the really big universal problems are picked off one by one there is indeed still plenty of room, for now at least, for custom code.</p> (By big universal problems I mean things like: accounting, payroll, Enterprise Resource Planning aka ERP, development tooling, website creation, online payments, social media, buying and selling privately online, car-sharing, short-term property rentals, travel bookings, etc. etc.)</p> Some organisations will always think they are too special or have too much ego invested in rolling their own (not-invented-here syndrome) to buy an off the shelf options. This is inefficient, but will happen nonetheless if they want to pay for it.</p> There will always be innovation and disruption, which will require engineering effort. Even in the age-old physical realm of engineering such as building roads, bridges and buildings there continues to be innovation and a need for skilled highway and bridge engineers.</p> Undeniably, however, there are whole classes of problems that software is capable of solving that have already been solved (some better than others of course), and a market of off the shelf or open source software dominates each particular problem leaving little room for expensive one-off custom builds to solve the same problem again.</p> As each vendor gets better at solving the problem at hand (or gets disrupted when they fail to), more and more use cases that used to require custom code are served by off-the-shelf software eliminating the need for developer time and replacing it with more-or-less need for installation, setup and configuration expertise.</p> The end-game for software development</h2> In the very long run (maybe another 100 years?) I see us getting to a steady state much like exists in traditional physical engineering. There are no vast new uncharted problem spaces waiting to return billions of dollars just for being first to get there. There will still be the occasional disruption but we'll look back on the dot-com boom in the late '90s just like we look back on the industrial revolution and the creation of railroads for the first time. We'll still have railroads, and we'll still have social media platforms and SaaS businesses; and sometimes one company will die and be replaced by others; but we won't be awestruck by silicon valley's ability to turn the world on it's head any more. It will just be normal.</p> As for the software developers and their trade; they won't go extinct, they'll probably have the same job titles more or less. There will be more developers working for the SaaS companies, infrastructure companies and boxed software companies (locally installed software will never die), and less developers working for businesses that aren't fundamentally software businesses. The latter will just buy the outputs of the software companies for far less than they would have had to pay a development team or outsourcer.</p> Developer salaries</h2> Companies that have money will always pay good money for good people, and software businesses serving worldwide business and consumer markets with software that costs nothing to replicate will always have plenty of money.</p> The "shortage" of software developers caused by the collapse in interest in computer science degrees after the dot-com bust combined with ever increasing production of software has already started to be mitigated as the "boot camp" model that has been running for a few years now at scale is starting to result in volumes of quality and sufficiently experienced engineers. This will push prices for the "average" developer down. In fact prices of the average developer haven't really moved much in the last 20 years in spite of the effects of inflation eating away at the real value of that income. £50-60k is still a reasonable salary for an experienced engineer in an average job in the UK just like it has been for years, while house prices have rocketed up in that time.</p> So in the long run apart from people that can make themselves invaluable to the richest companies (which takes more than being able to write Python or C#) will still be well rewarded, but the days of exceptional salaries for anyone that can write a for</code> loop let alone recursion are well behind us now.</p> The fractures and integrations between software</h2> An interesting effect of the ever-growing amount of coverage of off-the-shelf software and SaaS companies in the global problem-space of business is the rising challenges and expectations for making it all work together (often known as "systems integration").</p> For sectors where there are many similar businesses (think plumbers, accountants, travel agents), the software vendors will produce solutions that cover everything that the profession needs all in one offering (marketing, finance tracking, invoicing, customer contact etc.). They can do this cost-effectively by selling the same thing repeatedly for near-zero marginal cost.</p> Bigger more complicated businesses, or more unique businesses are left trying to avoid building everything they need from scratch and instead trying to cobble together a complete solution for their business from all the various SaaS and on-site offerings that are out there.</p> As businesses put solutions in place for each problem they have to solve, they then start looking at how well these things are working together, and mostly discover that they aren't. One SaaS thinks it has the answer to everything but is wrong, another on-site system thinks it owns the data about customers but the cloud-based CRM has got out of sync again and they keep upsetting customers by getting their facts wrong. So sometimes they just paper over the cracks with people based processes and spreadsheets. Sometimes they hire developers to try and glue systems together that really don't want to be glued together (through the magic of dubious quality APIs or maybe direct SQL database access).</p> Some bits of software provide value as a standalone thing (e.g. document or chat tools), but even with those there's a drive to integrate (e.g. single sign-on to avoid access for previous employees, auditing and storage for compliance etc). Other bits of software are very painful to leave disconnected (e.g. a storefront in Magento where customers purchase goods and an inventory ordering/management (ERP) system in SAP Business One that makes sure there's anything to ship).</p> To solve this latest self-inflicted pain of the software world a whole new class of software has emerged who's only job is to join systems together that don't know how to connect directly, or to act as an orchestrator of all the flows of data. You might have seen IFTTT (If This Then That) which is the consumer-centric version of this where you can make any number of events in one SaaS trigger actions elsewhere. Another is Business Process Automation (BPA) such as Codeless BPA which connect all sorts of enterprise software together by hook or by crook, while giving a central place to manage the flow of data and massage and monitor it on the way through as needed.</p> But even that</em> extra layer of connection still doesn't always give us the magical point and click connection we were promised decades ago where you can make your whole business work without talking to any nerds. For simpler cases you might get lucky, but as soon as you have to make a web hook work many people just glaze over and reach for the phone. And if something stops working between two bits of software that don't really like each other, and your business's whole income depends on that data flow, you're going to do whatever it takes to make it behave again. So in a slightly different shape we find ourselves yet again in need of a vast army of technical folks who really know what's going on and how to work the magic to make the wheels turn again and the ones and zeros flow like a spring down a mountain-side.</p> Service buses and event sourcing</h2> One of my tech friends mentioned this (very technical) approach to solving the pains. This is a really good point, as you can get better more scalable, more fault tolerant and more responsive systems with these approaches. The move to this approach is however contingent on either the vendors playing ball, or on your systems being sufficiently hand-coded already that you can make these kinds of architectural improvements. YMMV as they say. An interesting aside that's worth being aware of and considering. I won't go into what they are here as there's tons of content out there on this.</p> AI (Artificial Intelligence) and Machine Learning (ML)</h2> A hot topic of the day is AI and ML. These are growth areas for good reason as they can eliminate whole swathes of formerly labour intensive work, but personally I don't see programming in any meaningful sense as being one of them. I might be a bit biased of course. If anything this is likely to spawn yet more need for highly capably technical folks to build and integrate these new systems, regardless of where the boundaries of each system sits.</p> A new consultancy</h2> Yes there was a reason I was even thinking about this, let alone writing it down, but this isn't really a pitch; I genuinely find this stuff interesting to think and write about.</p> I'm building a new systems integration business off the back of my experience in tech, so if that's relevant to you drop me an email mailto:tim@timwise.co.uk</a> and let's talk. I'm looking to build my network of people who I/we can help, and also my network of people who might want to be involved in the new business.</p> A penny for your thoughts</h2> I'd love to talk to you on an actual phone call</em> if you thought this was interesting or relevant (or wrong or silly!)</p> Drop me a line on mailto:tim@timwise.co.uk</a> and arrange a call!</p> Footnotes</h2> Many names of non-custom software</h3> Sometimes you may hear software referred to as COTS, which is short for "Commercial Off The Shelf" software, or "boxed software". These days you might call that codeless, no-code or low-code. Or maybe "apps". The point here is that you don't have to pay developers to solve a problem (which is expensive, slow and error prone - good for developers paid for their time and skills, not so good for the business paying the bill).</p> Why I want you to use slack threads 2021-07-08T00:00:00+00:00 Someone has probably sent you a link to this blog post in response to you posting multiple messages in a slack channel on the same topic.</p> Don't worry, we're not angry with you, we just want to help. We also know that the slack interface can be a bit confusing, especially on mobile, and it's easy to accidentally respond in channel instead on a thread.</p> This post is not intended to shame or berate you, just to share information on how we can all make slack a nicer and less distracting thing to use for everyone.</p> </p> The above screenshot is an example of two consecutive messages in slack, this happens if you send the message in two parts. (Send one message, think of something extra, send another message.)</p> This post is here to explain to you with kindness why posting multiple messages like this might be causing issues for other people and offer some alternate approaches.</p> Why multiple messages are a problem for others</h2> It seems harmless to just post another couple of messages, right?</p> Slack in any active organisation can quickly become a firehose of information that can be quite hard to keep up with. It takes some effort from all involved to protect everyone's focus and flow time. As an aside if you want an insight into just how damaging distractions can be I recommend diving into Cal Newport's writings https://www.calnewport.com/books/</a>, particularly "Deep Work". Some of us have the luxury of turning off slack for most of the day, but some people need to keep up more actively than that.</p> Notifications and unread messages</h3> Slack allows you to carefully manage what notifications you receive, but for a channel it basically comes down to notifications on or off for channel messages.</p> If you have a channel that you want (or need) to keep up to date with in a timely manner then when someone sends a new channel message you will have intrusive notifications and as shown below an "unread channel" in bold-and-white sat glaring at you waiting to be read:</p> </p> If you use the convenient "All unread" feature then that will also sit glaring at you as below with an eye-catching "1 new message" button until you go and look:</p> </p> The problem with multiple related channel messages in this case is that if someone gets your first message, decides that it's not a conversation relevant to them and then goes back to trying to concentrate your second message will set off all their notifications once again and send them back to having unread messages to worry about.</p> If you had instead used the below alternatives of editing or threads then there would be no new notifications and no new unread messages when you edit your message or add to the thread.</p> Deciding what conversations to follow</h3> As someone who is trying to avoid slack-overwhelm the ability to follow and unfollow threads is a very useful feature. Slack will automatically give new notifications for threads where you have been mentioned with an @</code> or have added a message of your own to the thread.</p> To follow/unfollow a thread, hover over the top message in the thread, click the dots, and click "turn off notifications for replied" as shown below:</p> </p> If people are not correctly using threads to group themes of discussion then this feature becomes mostly useless.</p> As an aside this applies just as much to failure to start a new thread when the discussion changes topic as it does to failing to use threads in the first place.</p> Catching up with a channel</h3> Sometimes you will join a channel and then mute it (right-click, mute) because you want to be able to keep up to date on your own schedule, say daily or weekly.</p> </p> When people in the channel have conversations in the form of channel messages you can easily end up having to read 1000 messages to just know what's been going on for a day and catch up on anything important.</p> </p> If instead the people in the channel are disciplined in using threads then it can easily be down to 20 messages with some very long threads that you can dive into if you feel the need.</p> What you can do instead</h2> By using the following methods you can help everyone else in your slack channel make better use of their time, more easily follow relevant conversations, and have fewer distracting FOMO moments (Fear Of Missing Out).</p> Start a thread</h3> Start a thread on your own message and add more messages in there:</p> </p> This allows you to add further context etc for those interested without writing a massive message in one go directly in the channel or creating a string of independent channel messages.</p> </p> See also https://slack.com/intl/en-gb/help/articles/115000769927-Use-threads-to-organise-discussions-</a></p> Edit your existing message</h3> Did you just forget to mention something? You can edit the message you've already posted to make corrections or add more information.</p> Hover over the message and then press the "more actions" dots:</p> </p> </p> See also https://slack.com/intl/en-gb/help/articles/202395258-Edit-or-delete-messages</a></p> Delete your subsequent messages</h3> If you've already posted more than one channel message and then realise your mistake, you can delete the extra messages before converting to a thread or edit, leaving the channel nice and tidy for anyone who shows up later to read.</p> </p> This is particularly important to avoid ending up with multiple threads on different channel messages (on the same topic) as other people reply to different bits of your message resulting in confusion and a disjointed conversation.</p> </p> See also https://slack.com/intl/en-gb/help/articles/202395258-Edit-or-delete-messages</a></p> Offering gentle reminders with an emoji</h2> If your team knows to do this but forgets sometimes then add :start_a_thread:</code> as an emoji (under "customize workspace) to easily remind people when they forget.</p> </p> (Thread gif image source: https://slackmojis.com/</a>)</p> </p> Summary</h2> So in summary, please give a thought to others before you send a second message. It may be quick for you but if there are 50 people in the slack room then you are costing them much more time and attention than it saves you by not taking the time to construct a clean message or thread.</p> Further reading</h2> Equal Experts' slack guide</a></li> </ul> GitHub rebase and squash considered harmful 2021-03-15T00:00:00+00:00 Not to be confused with git rebase</code> and the ability to squash commits locally with interactive rebase, which is a whole other flamewar</a>.</p> </p> This post is predicated on giving a crap about the git history you and your team produce. If you don't care about good quality git history then you are wrong because git history is an important aspect of your engineering output</a>.</p> In short, "squash and merge" is never good, "rebase and merge" is almost never good, "create a merge commit" is a good default, ... and if you have a really great team then mainlining while pairing/mobbing can be great for mean-time-to-recovery but shouldn't be the only allowed way.</p> If you don't believe me yet, then let us now go reaaaally deep on all the details of all the ways you can do this and what makes them so good/bad/evil:</p> "Rebase/Squash and merge" - both bad</h2> Personally I dislike the "rebase</strong>" and "squash</strong>" buttons on GitHub because:</p> You don't find out what you actually put in main</code> until after it's done. If you don't believe me check the sha1 of the tip of your PR branch with the new tip of main</code>, your sha1 is nowhere to be seen. Github then has the gall to sign the commits as you if you've let it.</li> There is no record of which commit you based your branch on, so if there's a new incompatibility because main</code> has moved on there is no record of what commit your branch was based on to assist with figuring out where the problem was introduced, it just looks like a bad patch to main with a PR that could never have worked.</li> The generated commits lose the true commit metadata generated locally (authorship, timestamps, sha-1).</li> GitHub generates brand new commit(s) at the point of applying to main</code> with no chance to review first.</li> </ul> These two options behave in ways that are not immediately obvious (to be fair github improved the labels on the options after I first ranted about this, but it doesn't change the fundamental issues). And if you press one of them without realising what it's going to do, tough because it's already in main</code>, and if you have branch protection on (you probably should) then you can't even undo it; it's there foreeeeeeeever.</p> Someone reviewing your PR has no idea which of these three buttons you will push, which could have been the difference between approving (based on a tidy merge and nice branch history) and rejecting a PR (based on including junk history or a badly worded squash commit).</p> Let's dive in to the hell that github have created for us:</p> "Squash and merge" - pure evil</h3> This one takes all your carefully crafted commits (or pile of "wip" junk</a> if you are lazy or plan to squash) and combines them into a single commit, pausing to allow you to write a better commit message (which few people I've worked with bother to actually do, immortalizing "* wip" in the history forever) and then pushing it to main</code> without further opportunity to review what you are shipping.</p> The behaviour this option encourages, nay, mandates, is to create gigantic 100-file patches that make your eyes bleed to read, basically making git blame</code> useless beyond knowing who to hate.</p> Anyone who tries to create many small meaningful and logically coherent patches but who has to use this option has the faustian bargain of choosing between making their team review piles of small PRs (too much overhead) or generating one stinking commit in main</code> with all carefully crafted iterations lost to oblivion (for example commit 1: refactor everything, commit 2: make important one-line business change, commit to main: both at once, argh).</p> Note that github alters your original commit message to retro-fit PR numbers. Also evil.</p> If you make a change to a file that results in both renaming and modifying files such as a rename-refactor, you can make that easier to follow by doing it in two commits... unless you squash them together. This is exacerbated by git's rename detection</a> which is actually not stored at all, but instead done as a heuristic when viewing patches. If you split the file name change and contents change into two patches you can help a future reader follow what you did, but if you change a file too much and then squash it all together then git will treat it as add and delete, breaking the ability to follow a line of code back through history without even more detective work and guessing.</p> I have nothing good to say about the "Squash and merge" option. If you don't want to see the messy branches of your developers then use --first-parent</code></strong>. This option in github solves a non-problem and just makes everything worse. Anyone who thinks this option has any merit is wrong.</p> If an individual developer wants to create a single squashed commit for main</code> they should squash locally and PR that</strong>. This option adds precisely nothing to what is possible, encourages reviewing and shipping junk history and encourages bad behaviour and unreadable patches. If you think it annoys me you're right, it should never have been created. It enables shit developers to carry on being shit and generate less "noise" in the history. Why would you want a process that only exists to to take the edge off being around the worst developers?</p> "Rebase and merge" - 99% awful</h3> This option takes your commits, rebases them on main</code> and then fast-forwards main to the new top commit.</p> When this option has been used you can no longer tell by looking at the git history that there was ever a PR, which is actually useful context when looking back to work out why the hell something is how it is. (If you look at the commits in github it does magically show the PR number but you can't find that context any other way.)</p> This option is marginally less offensive to me if used judiciously by skilled teams. You might sensibly use it because your PR was only for peer review rather than because your commits deserve a logical grouping. If you can trust your team to intelligently decide when the commits in a PR would look nice as straight-line history in main</code> then perhaps leave this option available.</p> If you want your github workflow fool-proof and "scalable" then disable this option. Noobs will find this button and push it for the wrong reasons and give you a shit git history.</p> The upside of being clever occasionally is outweighed in my experience by the beautiful consistency of everything coming in as "merge PR NNN for feature YYY" and then being able to either ignore the details of the branch with --first-parent</code> (which you can't do if someone has used this button), or look at the commits in that PR as a logical group on the second-parent side when you need more granularity (easily done by running a first-parent log on the last commit of the branch before it was merged).</p> "Create a merge commit" aka "github flow" - good</h2> My preferred workflow is to use the normal "merge" button every time. This is sometimes know as "GitHub Flow</a>".</p> In my view you should generate your commits, sha-1 and all, exactly as they will be merged into main</code>, and then as a separate merge commit those should be combined into main</code>.</p> To avoid many "tramlines" do local rebases to avoid PR branches being based on very outdated main</code>.</p> This retains the commit exactly as you crafted it locally.</li> It calls out the difference between you writing your commit and deciding it can go into main</code>.</li> It retains the information about which commit you based your branch on.</li> </ul> There's something to be said for being able to do a git log --first-parent origin/main</code> and get a consistent list of merged PR branches. It's really quite readable compared to a mishmash of the different styles of merge to main</code>.</p> You can configure GitHub to disallow the other options if your team are on-board with the idea.</p> You could argue that single commit PRs are a lot of overhead for something trivial, but equally having to decide each time results in more time worrying about whether something justifies the special treatment of ending up directly on main (in GitHub's modified form), and getting it wrong sometimes.</p> If your PR branches are full of "wip", "fix tests" and other junk then the answer is not to throw this option out, the answer is to get better at creating good patch sets</strong> by learning to use git rebase --interactive</code> and thinking harder about what would make good patch before you start typing in your IDE. (And noticing when your patch is getting messy and stopping to reflect on good patch generation).</p> I'm aware there are counter-arguments, but in my experience on many teams this seems to end up being the cleanest balance of trade-offs. If you are an individual developer on your own project then pure mainline development can make more sense, though even then thinking of feature branches and having them explicitly merged where appropriate can neatly group related commits. The only compelling argument I've heard against this flow is mean-time-to-fix, but that doesn't mean you have to kill this approach entirely, just be able to use other approaches too.</p> But my interface can't do first-parent!</h2> Someone once complained at me that we shouldn't do merge commits because they make no sense when viewed in github's deficient history viewer, which shows a random mishmash of mainline and branch commits with no indication of which is which.</p> Seriously, use a better git viewer, there are literally thousands.</p> It 100% sucks that github, the flagship platform for git, still to this day does not have a proper branch view let alone first-parent view. Appalling. I'm even more horrified that GitHub desktop</a> has inherited this idiotic denial of branch based views. If the view of history / PRs in github doesn't show you what you need don't use it</strong>. Did you know you can review PRs by just pulling down the branch and looking at it locally? You are a capable programmer who uses many tools not an idiot that needs to be spoonfed powerpoint presentations.</p> Even visual studio can show you branches, merges and first-parent views, it's just a little tricky to find because they insisted on calling everything by different names and using mystery-meat buttons.</p> Sorry, but I don't have much sympathy with the "my tool is shit so we should do a worse job" complaint against doing things properly, it's weak.</p> The case against PRs</h2> The only team I have ever come across that (incorrectly) disabled the "create a merge commit" option did so (I believe) in the name of "mean-time-to-recovery". This is however a false dichotomy.</p> If you want fast "mean time to recovery" (MTTR) you need to be able to ship a patch to main</code> and on to production fast. There is nothing about the ability</em> to create merge commits or use PRs that prevents that. So long as you don't enable the "require a PR" option in github, there's nothing to stop a team pushing straight to main</code> when that's the right thing to do.</p> I like the idea that the requirement is not a pull request, but just two pairs of eyes on everything. That gives you the ability to be more responsive as pairs can now ship straight to production. That said, I just don't buy the idea of "mainline development" for every patch</a>. Some changes are just more complicated than a single patch and it's good to be able to break things down in a branch before merging. Don't get me wrong, I'm all for pulling groundwork out and mainlining it (if your dev team is good enough to do this) so that your final patch is smaller, but sometimes a feature change has to go in in one commit to main</code>, and without a branch plus a merge commit to main it is just not granular enough to make a useful history. And no, feature flags will not fix this for you, they are a useful thing for sure, and can reduce the size of unmerged branches, but sometimes you just have to change fundamental things that can't be "flagged".</p> Team culture is worth considering, regardless of tool configuration. If your team does everything by PR then when something is on fire they'll probably raise a PR and then sit around waiting for approval; and if your team always commits everything to main as they go they'll recover fast but their history will be full of mis-steps and fixups that could have been dealt with before they ever hit main. As always extremes are bad and anyone who fast-talks you into believing one extreme is a panacea is glossing over important nuance and interesting counter-examples.</p> Disabling merge commits doesn't prevent people from having slow PR based async processes, and equally, enabling merge commits and PRs doesn't stop you shipping a quick patch by pushing straight to main when something's on fire. The two things are orthogonal.</p> The Review part of PR</h2> The existence of a Pull Request implies a review by another developer before merging to main</code>. The series of commits, including their summary, description, author and gpg signature is something that we can review and provide feedback on as part of the review process in order to provide a high quality history for those that need to understand why code is how it is, sometimes years in the future.</p> I have personally heard many developers say "Don't worry about the crappy history I'm going to squash it". Note the future tense in here. So the reviewer is supposed to just approved the awful "wip" history, or ignore it entirely, and trust that the person that does the merge will pick the right button and generate a sane commit to main.</p> The branch history in the PR that was reviewed and approved is then thrown away and something completely different goes in to main without any opportunity for the reviewer to object or comment.</p> What is the point of reviewing something and then committing something completely different to main</code>?</p> The generated history does have value, should be of good quality, and peer-review is an important part of keeping that quality high.</p> Perfection?</h2> So what's the ideal? In my view it's the one of the following depending on the quality of your team and what your goals are:</p> High performing team with top-end developers - reducing mean-time-to-fix</h3> Two pairs of eyes rule.</li> Don't disable anything, just educate and trust (... apart from branch protection to avoid force-push to main because forward-only is good)</li> (...... and you can probably tell I wouldn't mind if you did disable squash and perhaps rebase given the above ranting).</li> </ul> </li> Use judgement for when to pair, mainline, PR etc.</li> Practice all approaches so that team is used to fast fixes.</li> </ul> Trust your team to use the right tool in the right moment. Big hairy feature? Break it down, use PRs if useful, ship as a merge commit with nice history and link to story and PR. Production on fire? Pair/mob and mainline that sucker.</p> Mediocre / mixed team, avoiding breakages over fast-fix</h3> Require PR for all changes.</li> Disable everything but "create merge commit".</li> Review PRs for quality of commit list and require rewriting branch history till it's up to team standards before merging.</li> </ul> Link your outlook calendar with your google calendar 2021-03-11T00:00:00+00:00 If you have a Google/GSuite calendar and a Microsoft Teams/Office365/Outlook calendar you can get the Microsoft Outlook calendar to pull in all the events from your Google calendar which is handy when you want to know if you have any gaps or conflicts.</p> Amazingly if you move google entries around in the outlook calendar they changes are pushed back to the google calendar.</p> Link the Google calendar into the Outlook calendar</h2> Go to your Teams calendar</li> Click the "add calendar" button</li> </ul> </a> Here you can see the "Add calendar" button as well as the end result of showing the two calendars together (google in red, outlook in blue)</p> Click "Add personal calendars" and follow the instructions. The below shows the result. (The calendar defaults to blue but we can change that).</li> </ul> </a></p> Give the Google calendar a different colour</h2> You can then give the calendar a different colour in "Edit my calendars"</p> </a></p> House buying and selling tip - make contact! 2021-03-04T00:00:00+00:00 I've learned a useful trick as part of buying and selling house(s) that I want to share with you just in case you find it useful. It could save you months of stress and delay, and maybe even save a deal that was going to fail.</p> The big secret</h2> Establish direct contact with the buyer/seller.</p> This will allow you to co-ordinate on making the sale happen and spot anywhere that the chain of communication has broken down. (E.g. both solicitors waiting for each other and a message between them that has vanished into the ether).</p> But how?</h2> This should be easy, but estate agents don't want you to go round them, and solicitors won't necessarily pass on your request. But there is a way round this.</p> Buying a house</h3> Simple, put a note through the seller's letterbox offering direct contact and including your contact details.</p> Here's a template note to a seller</a> that worked for me in a previous purchase.</p> Sample letter to sellers</h4> To: The Owners, [Address of house you want to buy]</p> [Your name here] [Your address (optional)]</p> Contact: [01184960000 / your.email@example.org]</p> [Date]</p> Hi,</p> We’re the people who put the offer in on your house, and we just wanted to give you the opportunity to make direct contact should you have any questions.</p> We still absolutely love your house, and are very keen to proceed with the purchase. We are working with [Estate Agent] to move the sale of our house forward.</p> Our contact details are above, don’t hesitate to get in touch should you wish to. We will of course still do all formal communications through the estate agents and solicitors, we’ve just found it handy in the past to also have direct contact.</p> Yours,</p> [Your name here]</p> </blockquote> Selling a house</h3> Write a formal letter with an offer of direct communications and include your contact details, PDF it and send it to your solicitor to pass on. Because this is officially a letter to the seller both solicitors will I think be obliged to pass it on.</p> Here's a template note to a buyer</a> that worked for me in a previous sale.</p> Sample letter to buyers</h4> FAO: [Buyer Name Here] C/O: [Buyer’s Solicitors] Ref: [Property Address] Thu 4 Mar 21</p> Dear [Name],</p> This is just a personal note in addition to the formalities around the sale to let you know that we are keen for the process to go smoothly for both sides and that if there’s anything you need from us you are welcome to make direct contact as needed.</p> All contractual and legal issues will continue to be dealt with through our solicitors and the estate agent, however we have found it useful in the past to be able to co-ordinate directly to ensure a smooth route to a completed sale.</p> There is no pressure to accept, but should you wish to do so please find my contact details below.</p> Confirmation of your receipt of this letter would be greatly appreciated.</p> Yours</p> [Your Name]</p> [your.email@example.org] [01184960000]</p> </blockquote> I ain't no lawyer</h2> Do this at your own risk! Just as this could help, you could also use this communication channel to sink a deal that would have worked out, or even get yourself in legal trouble.</p> Podcasting hardware setup 2021-01-26T00:00:00+00:00 Current recording/playback setup</h2> Samson Q2U cardiod XLR/USB Microphone</a> ~ £90 USB means no need for a mixer or XLR adapter thing.</li> Seems to be the best price/quality balance for me.</li> The cardiod effect is pretty good, greatly reducing input of noises from elsewhere, especially if you talk really close to it (which I don't).</li> </ul> </li> Proel RSM180 Microphone stand with boom</a> ~ £34 Because standing desk, plus the Samson mic doesn't isolate vibrations from the desk if it's on a desk mount.</li> </ul> </li> AKORD Microphone Swivel Pop Filter</a> ~ £5 Because all the cool kids have one, and it was cheap. No idea if I really need it. Edit: yes I do, breathing on the mics and hard "P"s really do create a bass boom without it as the rush of air hits the mic.</li> </ul> </li> Some old philips bluetooth headphones connected with a real headphone cable to the line out on the back of the Samson mic. I only use this for recording to avoid feedback from speakers. It has a feedback of your own voice which is disconcerting and nice in equal measure. For conference calls zoom and teams seem to do a decent job of cutting out the feedback, and I use mute a lot anyway so I don't bother with the headphones.</li> The tiny and highly capable Nobsound G3 2 Channel Bluetooth 5.0 Power Amplifier 100W</a> hifi amp, driving a pair of bookshelf Kefs for editing, music and conference calls.</li> Wharfedale Diamond SW150 Subwoofer</a> hiding at the back, because frequencies below 100Hz</a> are important too.</li> </ul> Video</h2> Camera</h3> Logitech C920 HD Pro Webcam</a> ~ £60</li> </ul> Lighting</h3> WBTY Gooseneck Mount Stand</a></li> Neewer USB lights x2</a> ~ £51 (didn't come with usb power units, bring your own, requires high-current capability)</li> </ul> Software</h2> OBS (Open Broadcaster Software)</a>for live streaming + recording video+audio</li> restream.io</a> for streaming to multiple platforms</li> Audacity for editing (I streamed a record+edit session</a> if you're interested)</li> </ul> Everything else</h2> Dell XPS15 (sometimes an XPS13)</li> Linux mint cinnamon</a></li> Standing desk</a> (no chair these days, previously kneely chairs and exercise balls).</li> Nextstand k2 Folding laptop stand</a> (amazing bit of kit, very portable too)</li> Microsoft sidewinder X4 keyboard</a></li> Logitech M570 trackball</a> - I find the continuous thumb+finger grip needed for a mouse uncomfortable. Recently upgraded to Logitech MX Ergo</a></li> </ul> </li> HP M477fdw laserjet + sheet feed scanner</a> because life's too short for manually scanning one page at a time. Configured to automatically send scans to server over smb, then sync'd to all devices with syncthing. Win.</li> </ul> Pics</h2> </p> Also-ran hardware</h2> I started with a Jabra Evolve 65 wireless headset</a> (~£130 because covid prices) but that heavily processes voice so while it's great for conference calls it's not good for anything beyond your third episode as people will unsubscribe just for the sound quality. Still use them for on the go conference calls and phone calls, and occasionally music (though my phone's bluetooth is currently borked). They eventually also snapped after being thrown in a bag too often, so now I have a...</p> Jabra Evolve2 65</a> headset, which I use for conference calls (and it's great for that), especially out and about, but which is also too heavily processed to be suitable for recording high quality podcast audio.</p> I tried a Blue Microphones Snowball USB Microphone</a> (~£75) next, but that picks up everything</em> even in it's alleged cardioid mode. It did make a good recording of some thunder until the kids talked over it in excitement! You could use this if you have a very</em> quiet environment.</p> Out of my price range</h2> Some of my friends who take it more seriously splashed out on a Shure SM7B XLR Mic</a> at a mere £389, and you have to by an XLR mixer to run it. I have to say they sound lurvely, and when they knock it you hardly hear it so it's got excellent vibration isolation built in.</p> You could also look at the Shure MV7 USB Microphone</a> which has USB as well as XLR so you wouldn't need a mixer. That's £250 currently. Presumably it's also got Shure's excellent sound and vibration isolation, though I haven't got one, so let me know if you get one!</p> See also</h2> Scott Hansleman's AV kit guide - "good, better, best"</a></li> </ul> A book list for my children 2021-01-25T00:00:00+00:00 This is for my children when they're old enough to read them and benefit from them, and when they decide to take an interest in the advice of their boring old man, however I'd recommend this list to anyone no matter their stage in life.</p> What book should you read next? Well, what problem are you trying to solve right now? That said, there are some eternal problems in life such as interpersonal relationships and the search for financial stability and fulfilment which these books help navigate.</p> I've added a note to each explaining why you should read it, and when you might want to make it a priority. I've also put them in an order that I think would be sensible; with the most foundational first (personal growth) followed by more wealth and lifestyle focussed volumes.</p> Being human</h2> The 7 habits of highly effective people; Stephen Covey</a> Why? It's important to know you can change who you are and be a better person, and know how to do so. You can be better with relationships and those around you. This book is an inspiration in personal growth.</li> When to read: as a young adult, and again when you're older and wiser. Or right now if you're already both of those.</li> </ul> </li> Daring Greatly by Brené Brown</a> Why? Our society teaches all the wrong lessons around guilt, shame and vulnerability. Our default responses to that are self-destructive. Brené shows us how to be brave enough to accept the challenges of life without hurting others to protect ourselves. The difference between guilt ("I did something that was wrong and regret it") and shame ("I'm a bad person") is often misunderstood or ignored entirely. When you think clearly about this it becomes clear that guilt is good and shame is bad.</li> When to read: As early as you can, and maybe again if you find yourself falling into the traps of self-shaming, shaming others or not having the courage to do hard things for fear of failure.</li> </ul> </li> Nonviolent Communication; Marshall Rosenberg</a> Why? Somewhat surprisingly perhaps the calm and empathic approach to communication is superior; but that's easier said than done with our brains evolved for times gone by.</li> When to read: As soon as you argue with someone and wonder what it achieved or it cost you dearly. Ideally before that happens.</li> </ul> </li> Bonds That Make Us Free; C. Terry Warner</a> Why? Yes, it's another book on dealing with people. The truth is it's not them it's you; but the good news is that you have the power to change what's wrong. It turns out that dealing with people is important, and is foundational in all walks of life.</li> When to read: As soon as you can, and then probably every decade after that.</li> </ul> </li> So good they can't ignore you; Cal Newport</a> Why? Don't believe the hype at work, people hire for your real skills and what you can do for them. More details and foundational career advice for any path you choose lie within.</li> When to read: Probably before your first or second job, but better late than never. If you've retired rich before you read this then maybe don't bother!</li> </ul> </li> The Life-Changing Magic of Tidying; Marie Kondo</a> Why? Because too much stuff ironically makes you less happy. Go figure. A practical guide to a calming and tidy house/room/boat/mansion/office from someone a little obsessed with tidy.</li> When to read: When you're fed up hearing me complain your room's a mess. Or later on when you realise why I complained about it but don't really know how to fix it.</li> </ul> </li> 48 Laws of Power; Robert Greene</a> Why? Sadly the world is not the cuddly, friendly, fair and equitable world some people seem to think they can wish into existence by will power and singing alone. There are plenty of good things, but the hard truths about raw underlying human nature are best understood so you can deal with them if you need to. I trust you are a good human being who will only use some of the darker knowledge in here for good and personal survival.</li> When to read: I hesitate to say early because used without experience this knowledge will get you into more trouble than it gets you out of. That said, earlier the better but don't rush into using it, just observe the people around quietly with this new knowledge for the next few years at least.</li> </ul> </li> Why Has Nobody Told Me This Before?; Julie Smith</a> Why? We all think we are perfect and we know how to use our brains. Turns out that's hubris, and it's worth having an expert on hand to help you learn to use your own brain properly and communicate in non-destructive ways with those around us. Expert therapy and counselling are great and should be taken advantage of, but it's useful to have a reference and guide to have and to study alone as well, regardless of whether you get expert help.</li> When to read: as soon as you can, and probably every 5-10 years after that given we tend to forget the lessons and go back to default human patterns.</li> </ul> </li> </ol> Money and work</h2> Rich dad, poor dad; Robert T. Kiyosaki</a> Why? Money isn't everything, but not having it will make you miserable. And the thing that will make you poor beyond anything else is your mindset and financial education. Read this to learn what the people who never seem to work do and why they never seem to run out of money.</li> When to read: As soon as your bank account hits zero for the first time, and any time you are annoyed that you still have that sucky job you hate but can't escape from (happens to us all I think).</li> </ul> </li> Think and Grow Rich; Napoleon Hill</a> Why? More money will let you solve more problems, have more freedom, help more people, and live your best life. But it's not that easy to know where to start without a role model and a plan. Good practical guidance in here. Time will tell if I make good use of it myself.</li> When to read: After the first few years of plying a trade perhaps, but don't leave it too long.</li> </ul> </li> The Millionaire Fast Lane; M.J. DeMarco</a> Why? Having read a lot of books about business and money, none of them have the completeness of this one. It's the best guide to the whole picture I've come across.</li> When to read: As soon as you start trying to earn your own money, or as soon after that as you can.</li> </ul> </li> The 4-Hour Work Week; Tim Ferris</a> Why? Working for someone else 9-5 isn't the only way to make a living. This book might age when it comes to tactics, but the inspirational vision of another way of life is timeless.</li> When to read: Whenever you've had enough of a job. Or maybe if you just want to dream, or travel.</li> </ul> </li> The Road Less Stupid; Keith J. Cunningham</a> Why? Whatever money and success you achieve, it's very easy to throw it all away again by doing dumb things that seemed like a good idea. This is a straight-talking guide to not doing that. Contains important lesson about protecting against downside risk (i.e. a small chance you lose everything) when chasing upside risk (i.e. a chance to be much richer etc). Unmitigated downside risk has the power to destroy everything you've worked for.</li> When to read: Sooner rather than later, especially if you're feeling a bit ambitious.</li> </ul> </li> Working Identity; Herminia Ibarra</a> Why? The days of job-for-life are long gone, even the days of one-career lives are fading. You'll almost certainly need to evolve your work life, if not blow it up completely and start over. This book is a wise guide to the opportunities and pitfalls of seeing if the grass really is greener.</li> When to read: when you are sick of your job/career and want to do something different.</li> </ul> </li> </ol> The meaning of life, evolution and religion</h2> The Blind Watchmaker; Richard Dawkins</a> Why? Evolution if often poorly understood. As the foundation of our existence and behaviours this to me is a must read to truly understand life. For me it answers the question of "what is the point in life"; ironically there is no point really, but that is more freeing than some deity with odd morals. Just know that you can know the raw mechanics and still live a full, happy and meaningful life; helping all those around you.</li> When to read: Whenever you have some time. Kinda background knowledge for life.</li> </ul> </li> The God Delusion; Richard Dawkins</a> Why? God is all around us in the many people's of the world. His name(s) are often used to manipulate and exploit, and play power games. You and I can probably do nothing about this (and it's dangerous territory to tread in, I'm surprised Dawkins is still alive), however I think the knowledge is an important piece of the complex puzzle that is humanity. Once you've read this, don't judge those who believe for whatever reason, you will not change them and you will only cause hurt.</li> When to read: Whenever you have time, any time you are considering the meaning and truth of religion, and any time you want to be able to discuss the topic with more useful information.</li> </ul> </li> </ol> Children of your own</h2> Pregnancy For Men</a> Why? Nothing can prepare you for this experience, and men are an anxious side-show. This book helped me cope, and even occasionally be useful.</li> When to read: As soon as you think you'll have a kid on the way, don't leave it till the due date or you'll miss the pre-natal stuff.</li> </ul> </li> How to talk so Kids Will listen</a> Why? Nothing about communicating well is natural or obvious. It takes work and understanding, and undoing cultural failings. This series is one of the best, kindest works out there, and helped me immensely.</li> When to read: When your kids are starting to talk.</li> </ul> </li> Hold on to Your Kids: Why Parents Need to Matter More Than Peers</a> Why? Culture is teaching us to ignore our parents with devastating consequences for all involved. It turns out "attachment" is the key, and when it goes wrong nothing you do will work.</li> When to read: When your kids are 4+ years old.</li> </ul> </li> </ol> Personally I like the audiobook format, and at time of writing a yearly subscription to Audible was good value for getting easy access to the content, and a narration from the author or a good speaker makes for a more memorable experience.</p> Beyond this list, check the best-seller lists for whatever subject you wish to excel at. There's no point learning the hard way when you can learn from someone who has. Even in this digital age books still pack a tremendous punch; the result of someone pouring 20+ years of their experience into something you can read in a week is like liquid gold. I also have recorded a lot of my personal reading list on GoodReads</a>.</p> To my little people who will soon be bigger than me in every way, love you 💕️👊️, Diddy x</p> P.S. If you're reading this and you're not my kid, well I love you too for being the lovely human I know you are.</p> Remote code interview pairing tools 2020-12-04T00:00:00+00:00 Things that you could use to do simple coding exercises in the new world of remote everything.</p> CodeInterview.io</a> - $5/interview or $55/month</li> CodeSandbox.io</a></li> Cyber-dojo</a> - free for non-commercial, set your own price for commercial!</li> VSCode live share</a> works well, it can do voice as well.</li> Zoom/hangouts/meet/skype etc. built-in screen sharing and whatever coding setup they already use.</li> </ul> Thanks to my friends at eSynergySolutions for the suggestions, recorded here because the internet has a better memory than me.</p> Database migration options for dotnet core 2020-11-26T00:00:00+00:00 Options</h2> Built-in EF Core migrations</a></li> FluentMigrator</a></li> DbUp</a></li> RedGate Deploy</a></li> RedGate SQL Change Automation</a></li> Flyway by RedGate</a></li> SqlHawk</a> (I wrote this years ago, forked from the excellent SchemaSpy</a>)</li> ? hit me up on twitter</a> if you know of any more options</li> </ul> Note that the excellent ReadyRoll got bought by RedGate</a> and renamed to something I can never remember.</p> See also</h2> https://stackoverflow.com/questions/46649238/alternative-to-redgate-readyroll-yet</a></li> </ul> Running the CosmosDB emulator on Linux 2020-09-17T00:00:00+00:00 Update: microsoft have released a linux compatible docker image it seems, let me know if it actually works and I'll update this post. https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator</a></p> Unfortunately even though "Microsoft 🖤 Linux" the CosmosDB team seem less keen. This is not fun, pretty or stable, but at least it works which is something.</p> There is a docker image but it only runs on docker-for-windows so it's no help here.</p> Overview</h2> Grab a Windows VM, install the emulator.</li> Copy the emulator SSL cert to the host.</li> Map the ports.</li> Redirect the VM IP back to localhost with iptables.</li> </ol> Steps</h2> Get a Windows VirtualBox VM Running</h3> Download the VirtualBox image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/</a> (2 month free trial, no key needed).</p> Install VirtualBox.</p> Fix the VM image configuration (if you are using the slightly older VirtualBox from ubuntu's apt repositories) by removing controller="VBoxSVGA"</code> from the .ovf</code> file in the .ova</code> file (which is a tar.gz archive). See https://schnouki.net/post/2020/how-to-run-a-win10-dev-vm-on-virtualbox-5/</a> for details.</p> Import the VM into VirtualBox</p> Run the VM</p> Install the guest additions (Devices > Insert guest additions CD)</p> Set the timezone (VM defaults to US).</p> Turn on seamless copy-paste & drag-and-drop (so that we can move certificate and data files to/from the host easily).</p> Set up the CosmosDB emulator</h3> In the Windows VM download and install the CosmosDB emulator in the VM from https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator?tabs=ssl-netstd21#installation</a></p> If the emulator is already running, shut it down:</p> cd </span>"</span>C:\Program Files\Azure Cosmos DB Emulator</span>"</span></span> .\</span>Microsoft.Azure.Cosmos.Emulator.exe</span> /</span>shutdown</span></span></code></pre> Start the emulator with network access enabled.</p> cd </span>"</span>C:\Program Files\Azure Cosmos DB Emulator</span>"</span></span> .\</span>Microsoft.Azure.Cosmos.Emulator.exe</span> /</span>AllowNetworkAccess </span>/</span>DisableRateLimiting </span>/</span>Key</span>=</span>'</span>C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==</span>'</span></span></code></pre> (This key is the hard-coded well-known key that the emulator uses by default, which makes it easy to connect the storage explorer).</p> Check it is listening on 0.0.0.0:8081</code> (as opposed to 127.0.0.1:8081</code>):</p> netstat -a \|findstr 8081</span></span></code></pre>Map the CosmosDB port to the host</h3> In the VirtualBox network settings for the VM, click advanced and add a port mapping for 8081 on the VM to 8081 on the host.</p> See https://www.howtogeek.com/122641/how-to-forward-ports-to-a-virtual-machine-and-use-it-as-a-server/</a></p> This makes the emulator appear as if it was running on the Linux host.</p> (Note there are replica ports that need mapping too, see the dotnet-core section below).</p> Connect with Storage Explorer</h3> Install the Azure Storage Explorer on your Linux host, either with the .tar.gz or from the snap store</p> https://azure.microsoft.com/en-us/features/storage-explorer/</a></p> Run the storage explorer.</p> Connect to the emulator on localhost.</p> You should now be able to create collections and add documents, and then view them.</p> Connecting from dotnet-core</h3> Problems connecting from core that I didn't encounter with storage explorer:</p> The SSL certificate is rejected.</li> CosmosDB sets the ReadEndpoint</code></a> to be the VM's IP address, which isn't accessible from the host because we are using NAT and port mapping. (oddly this wasn't consistently the interal IP address.</li> The replicas listen on additional ports which also need mapping.</li> </ol> Accepting the emulator SSL certificate</h4> This is the error you get trying to connect to the CosmosDB emulator from dotnet core on Linux without adding the certificate to the Linux host's certificate authority (CA) list:</p> System.Net.Http.HttpRequestException:</span></span> The SSL connection could not be established,</span></span> see inner exception. ---> System.Security.Authentication.AuthenticationException:</span></span> The remote certificate is invalid according to the validation procedure.</span></span> at at System.Net.Security.SslStream.StartSendAuthResetSignal(</span></span> ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)</span></span></code></pre> Export the emulator's public SSL certificate "DocumentDbEmulatorCertificate" as base64 X509.</p> Reference: https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator-export-ssl-certificates#how-to-export-the-azure-cosmos-db-tlsssl-certificate</a></p> Copy the exported file to your Linux host.</p> Add the certificate to the trusted certificates on your machine (these instructions are for Ubuntu/Mint):</p> sudo cp --verbose cosmos-emulator.cer /usr/local/share/ca-certificates/cosmos-emulator.crt</span></span> sudo update-ca-certificates --verbose</span></span></code></pre> Reference: https://stackoverflow.com/questions/44159793/trusted-root-certificates-in-dotnet-core-on-linux-rhel-7-1/44160125#44160125</a></p> Note the change in file extension from .cer</code> to .crt</code> between export and import (I'm not 100% sure if this matters).</p> Note that the certificate is regenerated on emulator startup</em> so you'll either have to do this every time or pass the emulator a predefined certificate to use.</p> If you want to validate the certificate has been installed, see https://unix.stackexchange.com/questions/97244/list-all-available-ssl-ca-certificates</a></p> Redirecting ReadEndpoint traffic</h4> In the VM in a powershell or command prompt run ipconfig</code> to get the machine's IP address.</p> Tell iptables</code> to redirect all requests for the VM's IP address back to localhost</code>:</p> sudo</span> iptables</span> -</span>t</span> nat</span> -</span>I</span> OUTPUT</span> -</span>-dst</span> 10.0.2.XXX</span> -</span>p</span> tcp</span> -</span>-dport</span> 8081</span> -</span>j</span> REDIRECT</span> -</span>-to-ports</span> 8081</span></span></code></pre> Where XXX</code> is the IP of your VM.</p> As per: https://unix.stackexchange.com/questions/441182/how-to-map-an-ip-address-to-localhost</a></p> Mapping replica port(s)</h4> On the VM run netstat -a \|findstr LISTENING</code></p> You'll see a bunch of ports in the 10,000-20,000 range. These are the replicas</p> I found out which ports to map by running the client code and looking at the exception when it failed to connect. It was only port 10253</code> for me so far. Here's the error for a failure:</p> Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.</span></span> ActivityId: 2dfaf5dc-0d5a-4ed7-828d-c16cdcd117a3, </span></span> RequestStartTime: 2020-09-17T20:00:16.8868999Z, RequestEndTime: 2020-09-17T20:00:46.7777188Z, Number of regions attempted: 1</span></span> ResponseTime: 2020-09-17T20:00:16.9400284Z, StoreResult: StorePhysicalAddress: rntbd://10.0.2.15:10253/apps/DocDbApp/services/DocDbServer15/partitions/a4cb495b-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-09-17T20:00:16.9221735Z, activity ID: 2dfaf5dc-0d5a-4ed7-828d-c16cdcd117a3, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D], URI: rntbd://10.0.2.15:10253/, connection: <not connected> -> rntbd://10.0.2.15:10253/, payload sent: False, CPU history: not available, CPU count: 8), ResourceType: Document, OperationType: Query</span></span></code></pre> Add these ports to the VirtualBox port mapping configuration for your VM.</p> In the VM in a powershell or command prompt run ipconfig</code> to get the machine's IP address.</p> Tell iptables</code> to redirect all requests for the VM's IP address back to localhost</code>:</p> sudo</span> iptables</span> -</span>t</span> nat</span> -</span>I</span> OUTPUT</span> -</span>-dst</span> 10.0.2.XXX</span> -</span>p</span> tcp</span> -</span>-dport</span> 10253</span> -</span>j</span> REDIRECT</span> -</span>-to-ports</span> 10253</span></span></code></pre> Where XXX</code> is the IP of your VM and 10253</code> is the replica port.</p> As per: https://unix.stackexchange.com/questions/441182/how-to-map-an-ip-address-to-localhost</a></p> Importing data</h2> If you need to import data then use data migration tool from https://docs.microsoft.com/en-us/azure/cosmos-db/import-data#Install</a></p> Success</h2> With all of the above I was then able to connect a dotnet-core application on Linux to a CosmosDB emulator.</p> Emulator failures</h2> After a while and some unrecorded fiddling, the emulator then gave up the ghost, resulting in the Azure Storage Explorer reporting the apparently meaningless error message "Unable to retrieve child resources."</p> According to the internet this error could mean anything. https://social.technet.microsoft.com/wiki/contents/articles/53393.azure-storage-explorer-troubleshooting-unable-to-retrieve-child-resources-or-the-request-action-could-not-be-completed.aspx</a></p> I might just blow the whole VM away and start again...</p> Should you use CosmosDB?</h2> CosmosDB comes with significantly increased costs:</p> I gather it's very expensive to run in production</li> Development is painful</li> </ul> As a result you should only choose CosmosDB for your solution if you really</em> need something it offers that you can't get from a more straight-forward SQL or DocumentDB solution and your domain is a good match for the document-database (aka NoSQL) model.</p> Further reading</h2> Azure Development using CosmosDB offline Emulator by Matt Thankachan</a></li> </ul> Effective GTD with Trello 2020-09-15T00:00:00+00:00 Learn how I've refined the GTD process with Trello for modern living.</li> Get a template GTD board to kick-start your own.</li> </ul> GTD+Trello has really helped sooth my neurotic tendencies around todo lists. I'm often startled by starting off thinking I have more than I could ever achieve and then getting all my top priorities done in one sitting and being able to truly relax afterwards. No more nagging feeling there's something urgent at the bottom of the pile of papers somewhere.</p> What is GTD?</h2> Getting Things Done https://en.wikipedia.org/wiki/Getting_Things_Done</a> is an old but effective method for organising the flood of todos that enter your life in many forms. We need it now more than ever in our infinitely connected world. And as a middle-aged working parent I need it now more than I ever did.</p> The point of GTD is to eliminate all the other places that you might have hidden todo lists such as</p> unread emails</li> your memory</li> phone notes</li> bits of paper</li> phone screenshots</li> other unstructured todo lists (Trello, Evernote etc. let me know what you use now that isn't working for you and I'll add it to the list)</li> </ul> And then to make your todo list something that you can almost mindlessly churn through, eliminating huge task switching costs caused by badly defined and variably sized tasks.</p> Trello + GTD</h2> The original GTD method involved bits of paper. But now we have Trello, which is an excellent fit for GTD with its cards, lists and many integrations. There's a great write-up of using Trello for GTD here: https://blog.trello.com/gtd-getting-things-done-maximizing-productivity-trello</a> which I suggest you go and read first. It's how I got started, however I found I was still left building my own process and set of lists from there before it really worked for me.</p> Trello is great for GTD because:</p> You can set up lists how you like</li> There are apps so you have it everywhere</li> You can forward emails to it (yay, inbox zero at last)</li> It supports attaching pictures, PDFs etc.</li> You can add checklists, due dates, date-stamped comments etc.</li> Drag-n-drop sorting and moving of cards</li> Buttons for archive / move on the cards</li> A lovely cross-platform interface</li> It's free!</li> </ul> Disclaimer</h2> Neither GTD or Trello can give you more hours in the day, help you focus, get you to do things you really don't want to do, or make your tax return go away. This still requires putting the work in. Having said that, at least you will know you haven't just forgotten anything really important, it's just a bit stuck on your list. Personally with life as it is for me now I never get close to the weekly/daily GTD schedule laid out below, but that doesn't make it less valuable.</p> If you are looking at GTD for the same reasons as I did then beware "the efficiency trap"</a></p> My GTD Method</h2> Here's my actual GTD board (minus details) to give you an idea of what it looks like. You might be able to see I have far too much in Inbox and Action items as is often the case, but it doesn't take too long to clear down when I actually focus on it.</p> </p> First you need to get everything you might have to do into your GTD board. Don't think too hard at this point, just ping everything to Trello and deal with it efficiently at your next GTD moment:</p> Getting Everything to the Board</h3> Emails</h4> Forward emails to your GTD board "send-to-board" email address to get to inbox zero quickly. Make sure your GTD board email address is in your address book so you can just type "gtd" and hit send.</p> </p> Using GTD instead of your unread email as a todo list makes inbox-zero possible again.</p> The subject line becomes the card title so edit that if you want. The body of the email becomes the card description so you can add to that too.</p> Top tip: When sending an email that you'll need to follow up on, put the GTD email address in the Blind-Carbon-Copy (BCC) field so that you instantly get a copy in your GTD board with no extra faff. (Thanks to this podcast episode for that idea: "Organizing the LEGOs In Our Minds to Build the Ultimate Version of Ourselves with Tiago Forte - Flavor of Leadership"</a>).</p> Web based tasks</h4> Install AddToAny</a> to send any webpage from Firefox to your GTD inbox.</p> </p> Documents, physical objects, pictures, screenshots</h4> Install the Trello app and use the native share buttons to share pictures, web pages etc to your GTD inbox. Use this to take photos of paper documents and physical things you need to deal with and send the picture straight to your GTD inbox on Trello.</p> </p> Verbally Received Tasks</h4> For when someone just says to you "oh, can you do this for me?" and you don't want to be that person that lets them down by forgetting.</p> If you think you are "the forgetful type" (you aren't, that's just a negative self-label compounded by habits), then you can fix it forever by just being more organised, no-one will ever know it was GTD, they just think you remembered for them.</p> Open up Trello on your phone or laptop and add a card. Or if it's easier write an email to your GTD board!</p> Using Alexa/Siri etc</h4> For advanced magic teach your Siri/Alexa/Google to email to GTD by mere voice command</p> Hey google, email gtd ...</p> </blockquote> Or maybe set up a custom action. (If anyone's done this send me some details and I'll expand out this section. PRs welcome!)</p> GTDing</h3> Inbox Processing (Daily)</h4> Go through inbox cards and triage them as follows:</p> Unactionable or not important? Archive the card</li> or move to reference</li> or move to someday.</li> </ul> </li> Less than 2 mins? Do now.</li> </ul> </li> For someone else? Move to "waiting"</li> </ul> </li> Need to just mull it over a bit? Move to "incubate"</li> </ul> </li> For future time/date? Add to google calendar</a> as event or timed reminder and move to "scheduled".</li> </ul> </li> More than single step to complete? (Even two really small steps counts) Move to "projects" and add a checklist.</li> For some multi-step things I keep it as a single card, put it in actions but put the next step in the title (e.g. "something to do - call bob"), and when I've done that step I update the title to the next step. It's a bit less overhead than juggling multiple cards, but it's a judgement call whether it's worth a project card. I like project cards for things that are going to drag on over months.</li> </ul> </li> </ul> </li> Finally if none of the above then: move to "action items" and edit the card to make sure it's doable in one go (add links, forms, phone numbers etc).</li> </ul> </li> </ul> </li> </ul> I've learned to do more of this in the Trello app when I have five idle minutes during the day so that when I sit down at the laptop I don't use all the time just getting GTD straight.</p> I add some Trello labels to the cards so that I can tell whether a card is something I can pick up in any particular situation (do I need the laptop? good phone signal? good internet? do I need to be somewhere in particular?) The situational ones are all black with an @</code> prefix, e.g. @home</code> or @good-internet</code>.</p> Projects (Weekly)</h4> Go through each project</p> Do I still need this project (move to someday, or just archive the card if not).</li> Create next action card - use the "convert checklist item to card" feature of Trello.</li> </ul> Waiting (Weekly)</h4> Set due dates on the cards so that they'll get highlighted if they are due chasing next time you review.</li> Chase up if no action from others and past the due date (maybe fire off an email or WhatsApp).</li> Archive if dealt with, or move back to inbox if it's thrown up other issues.</li> </ul> Rules - Thou Shall Not Scroll</h4> I.e. things that I've learned knock me off track when trying to do a "GTD" session.</p> DO NOT READ the articles - send them to Pocket</a>.</li> DO NOT READ the newsletters - leave them as unread in mail inbox.</li> NO twitter/facebook/linkedin FEED reading (posting is okay).</li> </ul> Dealing With Action Items</h3> Once your GTD board is triaged your inbox should be empty and your action items should be prioritized and full of things that you can "just do" without getting blocked. Start at the top and work down as if your boss told you to get on with it.</p> This is where the magic happens!</p> What to do when "Action Items" is too long or not getting done</h3> I found that after a while Action items gets overwhelming. Cards end up stuck in there forever, never quite getting to done.</p> To deal with this, use the "move all cards in list feature of Trello".</p> You have two choices here depending on the state of your list and how important thing things in it are:</p> Move all cards back to Inbox and re-triage them.</li> Move all the cards to "Someday" because they actually weren't that important.</li> </ul> Or you might pick a few to bump back to inbox then move everything else.</p> This gives you a complete reset, and eliminates the mental overwhelm that results from an unrealistic todo list.</p> You might initially think this is a failure of the GTD method, but actually without GTD you are just forgetting things that have been on your brain's todo list, and maybe some of them were actually important. With this approach you have the opportunity to decide what you want to punt to the eternal pot of "someday" with no angst that you are dropping important tasks.</p> If you have tasks that really are important and they are never getting done, then</p> you're avoiding it and you need to move them to the top and just do them, a well formed action card is often not as bad when you actually tackle it,</li> or you don't allocate sufficient time in your life for these tasks, and it's up to you if you want to adjust your schedule to get more done.</li> </ul> Watch Out For</h3> Spending all your time just arranging your GTD board and never doing any of the action items. Are you avoiding an unpleasant task? Are your tasks too big and need breaking down?</p> When the lists get long</h3> You have to make sure you're investing enough time on average for the piles not to build up and up.</p> When my list got to the several hundred range I needed to know for my own sanity whether I was winning or losing week by week, it's hard to remember whether the inbox was on 197 or 185 the week before. Being a bit of a systems person not afraid of a spreadsheet, I enabled the firefox trello card counter addon</a>, manually copied them into a spreadsheet, added a graph, and now I can see if I need to spend more or less time getting on top of it.</p> </p> </p> I've shared a google sheets copy of the card count graph spreadsheet</a> which you can use yourself.</p> Learn more</h2> Ready for Anything Book</h3> David Allen wrote a follow-up book with a huge amount of wisdom and experience embedded in it. This one is like having a management consultant in your ear guiding you to be the CEO of your life and work. Amazing. Not to be missed once you have the basics down.</p> "Ready for Anything: 52 Productivity Principles for Work and Life" by David Allen</a></p> Podcast</h3> Having run this system for a while, it grew to a daunting few-hundred cards in inbox/actions. To figure out what I'm doing wrong I did some more research.</p> The official GTD podcast</a> is gold, hearing from the creator and his consultants on how they really use it and help clients get unstuck was really helpful.</p> Key points I learned:</p> Projects are not what you'd normally think of, they're just anything bigger than one action. Anything</em>. (You can add another layer for big projects if that's your world).</li> The weekly review isn't just shuffling your todo pile. It's stepping back and considering the big picture, setting strategic direction, and just having time to reflect.</li> If tasks are getting stuck, or you are avoiding them it's probably because you haven't taken the time to step back and think clearly about whether it's something you actually want to do (someday/incubate/delete), and if you do then thinking about what the one next step might be (e.g. "google xx to learn more" about it).</li> </ol> Articles</h3> In no particular order, and of varying usefulness:</p> https://facilethings.com/blog/en/starting-gtd-when-you-are-busy</a></li> https://lifehacker.com/the-weekly-review-how-one-hour-can-save-you-a-week-s-w-5908816</a></li> https://todoist.com/productivity-methods/getting-things-done</a></li> https://blog.doist.com/gtd-tips/</a></li> https://www.reddit.com/r/gtd/comments/8bv3as/how_do_i_keep_my_gtd_inbox_from_overflowing_to/</a></li> https://plan.io/blog/getting-things-done/</a></li> https://www.taskade.com/blog/outliner-note-taking/</a></li> https://www.taskade.com/blog/getting-things-done-gtd/</a></li> https://www.taskade.com/blog/hierarchical-note-taking/</a></li> https://zapier.com/blog/gtd-getting-things-done/</a></li> </ul> Get started with a template GTD+Trello board</h2> You can easily create your own GTD Trello board from this template</a> that I've created for you.</p> The template contains:</p> Preconfigured columns,</li> Labels,</li> Some example cards,</li> A full instruction list in the inbox for daily processing,</li> A butler action button for moving to "someday".</li> </ul> I reckon using this ready-made template will save you at least an half-an-hour of your time just fiddling with Trello to get going. It'd also be a nice way of saying thanks if you found this article useful.</p> </p> Beyond GTD</h2> I still struggle with what to do today</em> even with GTD in place. I've found the (paid) Sunsama</a> app to be a game changer, especially as I transition to full business owner from sole contractor. You could do it without, but having an app that continuously reminds you what you planned to do today & this week makes sticking to priorities so much easier.</p> </p> Approaches to refactoring, technical debt and legacy code 2020-07-09T00:00:00+00:00 Sometimes a codebase has an overwhelming amount of "terrible" stuff that as a developer you almost can't help but just diving in and fixing it. Doing this without thinking too hard can result in many variations of failure, such as:</p> Upsetting the people paying you because you are doing something they don't consider important.</li> Never finishing the mammoth rewrite you took on, resulting in a mess of two different styles of code.</li> Personal burnout.</li> Fixing things that maybe weren't actually that important (even if they make you cry whenever you have to look at their source code), at the expense of failing to fix more urgent problems or building useful features.</li> </ul> This article is a collection of research, links, thoughts and ideas that I've captured here for both of our benefits. It's a bit disjointed to read, but the point was more to capture all the key concepts and resources that are floating around in tech land.</p> ⏩ If you want the executive summary, head over to my Leaders guide to technical debt - aka "why can't we ship anything!?"</a> 🧐. For the technical details read on below.</p> </p> Must-read articles</h2> Here's a couple of must-read articles on the subject:</p> Refactoring</h3> "We take the next feature that we are asked to build, and instead of detouring around all the weeds and bushes, we take the time to clear a path through some of them."</p> ~ Ron Jeffries</p> </blockquote> In short, don't make debt cleanup its own task, just tackle what gets in your way as you go, and leave the camp tidier than you found it.</p> I recommend reading the whole article here: Refactoring -- Not on the backlog! by Ron Jeffries</a>, it's not too long.</p> Technical Debt</h3> "Tech debt": an overloaded term. There are at least 5 distinct things we mean [when] we say “technical debt”.</p> Maintenance work</li> Features of the codebase that resist change</li> Operability choices that resist change</li> Code choices that suck the will to live</li> Dependencies that resist upgrading</li> </ol> </blockquote> Read the full explanation of each type of debt here: Towards an understanding of technical debt by Kellan Elliott-McCrea</a>, a bit longer but an important piece of writing.</p> Huge piles of debt</h2> If your project is sooo "bad" that you feel like throwing it out, you might do well to heed this quote:</p> "There is only one way to eat an elephant: a bite at a time."</p> ~ Desmond Tutu (via Psychology today of all places!)</a></p> </blockquote> Which to me means iterate your way to good.</p> It doesn't mean have no plan, in fact you should know where you are trying to get to and how realistic that is so you don't end up with a rewrite that can never be completed because it was too ambitious.</p> Even more reading</h2> Is a mess a debt?</h3> "A mess is not a technical debt. A mess is just a mess."</p> ~ Uncle Bob</p> </blockquote> A Mess is not a Technical Debt by Uncle Bob</a> suggests that the use of the term debt is for a considered short term trade-off just like taking out a loan, but a mess is nothing of the sort as there is no up-side to creating a mess versus having a clean but temporary solution to a problem.</p> More taxonomy of bad code: the Reckless/Prudent vs Deliberate/Indavertent quadrant</h3> Martin Fowler shows that you can categorize debt by whether it is reckless or prudent, and separately whether it was deliberately or inadvertently added to the code.</p> "Technical Debt is a metaphor, so the real question is whether or not the debt metaphor is helpful about thinking about how to deal with design problems, and how to communicate that thinking. A particular benefit of the debt metaphor is that it's very handy for communicating to non-technical people."</p> "The useful distinction isn't between debt or non-debt, but between prudent and reckless debt ... there's also a difference between deliberate and inadvertent debt."</p> "The decision of paying the interest versus paying down the principal still applies, so the metaphor is still helpful for this case."</p> ~ The Technical Debt Quadrant by Martin Fowler</a></p> </blockquote> These are useful definitions that can help clearly communicate complex issues with the code to people not directly involved in the code.</p> It's well worth reading much more of Martin Fowler's writing. There's so much to learn from Martin about programming good practice; the articles are all written in a very human and accessible style.</p> Legacy tests</h3> Now that TDD is widely adopted we're faced with cleaning up the output of those who cargo-culted</a> test coverage, or had well meaning but ultimately badly formed attempts at adding tests to their code, sometimes with prodigious volumes of test code.</p> I think "Legacy tests" is a useful term to describe these problematic tests. Nat Pryce outlines some useful hints that you have legacy tests on your hands:</p> "Symptoms of legacy tests I have encountered include: ... Tests are named after issue identifiers in the company's issue tracker. Bonus points if the issue tracker no longer exists."</p> ~ Nat Pryce, Working Effectively with Legacy Tests</a></p> </blockquote> Prevention is better than cure</h2> How do we stop it building up in the first place?</p> I see several sources of the build up of technical debt:</p> Lack of knowledge and experiencd of programming patterns and practices to keep debt in check. This can be addressed with courses, learning from peers, attending meetups and conferences, reading books, watching youtube and many other ways of getting input from beyond the current code editor.</li> Problematic mindsets in programmers - short term thinking, narrow thinking, or blindness to the broader consequences of the choices being made in the moment. This is a more stubborn malaise, and people are not quick to change their identity. Some will respond well to just being asked to consider the second and third order effects of their choices (e.g. "you could give this other team direct access to the sql db instead building an API for them, and yes it's quicker, but then you won't ever again be able to rename a column without consulting with them), but some people will double down in their thinking and are not open to improvement.</li> Dysfunctional dynamics between engineering and their non-technical peers and leaders. Common issues include: Deadline-driven-development (especially when unrealistic and vigorously enforced),</li> Asking non-technical leaders to make engineering decisions.</li> </ol> </li> </ol> The client / boss / project manager responds [whilst thinking “I have no idea what you are talking about, or why you are asking] “Which is quicker? Do that one.”</p> Then the engineer gets the hump that they can’t do a good job.</p> ~ from "Why do automated tests matter?" by Tim Abell</a></p> </blockquote> The audio version from Codurance</h2> Codurance</a> hosted an insightful round-table podcast episode with a group of people who are clearly very experienced. You can listen here: https://codurance.com/podcasts/2019-01-21-legacy-code/</a> and will doubtless be inspired by some things in there. The conversation takes a little while to build momentum but it's worth the wait.</p> Here some highlights of what I learnt from listening to the show:</p> The Feathers Book</h3> Working Effectively with Legacy Code by Michael C. Feathers</a> is the</em> book to read on the subject.</li> </ul> Feathers defines legacy code as code without tests.</p> "Preserving behaviour is a large challenge. When we need to make changes and preserve behaviour, it can involve considerable risk."</p> ~ Michael C. Feathers</p> </blockquote> This book is heavily geared towards unpicking the difficulties in getting untested code under test in object-oriented static typed languages such as Java, C# and C++, and modifying it safely.</p> It is a detailed and thorough treatment of all the tactics available to bring untested object-oriented code under test, specifically sprawling and smelly classes and methods. It won't help you with larger scale problems such as poor structuring of microservice architecture, message bus systems etc.</p> This article summarizing the Feathers' Legacy Code book</a> has some of the tactics detailed and gives a good flavour of the book so you can decide whether to dive deeper.</p> Named techniques and related libraries</h3> Characterization testing</a> is the idea of creating tests to probe and demonstrate the existing behaviour of previously untested code. Related to this is "approval tests" which allow you to easily incorporate snapshots of output (json, xml, logs etc) into your tests in order to capture existing behaviour and be able to spot any variations that pop up during refactoring.</li> ApprovalTests.net</a> is a dotnet library for implementing approval tests.</li> </ul> </li> Introducing seams</a> into software can be a useful technique for breaking down untestable monoliths into testable chunks on the way to better code.</li> Mutation testing (wikipedia)</a> (more info on mutation testing at csharp academy</a>) is a useful way of checking how good your test coverage really is. It is the idea of making (almost) random changes to the code under test to see what whether your tests spot the change in behaviour. For dotnet this can be done with Stryker.net</a></li> </ul> </li> </ul> Approaches from hard-won experience</h3> Make as few changes as possible to get untested production code under test. The first cut of tests will likely be fragile.</li> It's more important that legacy code that is already in production continues to behave as it currently does than that it behaves as originally specified. People and downstream systems may now rely on that "incorrect" behaviour.</li> Does the organisation (culture, systems, pressures etc.) cause bad code to be created? If you don't fix that then you will always get more "legacy" code.</li> The importance of competent technical leadership within an organisation for preventing the build up of catastrophic levels of technical debt.</li> When communicating, quantify the cost of problems with the legacy code. E.g. "you are losing 1 in 5 developer-days to coping with bugs introduced due to the lack of automated regression tests".</li> Have the hard conversations with the business about the cost of fixing the mess.</li> Doing a rewrite is (almost) always the wrong answer.</li> Get small wins, even if you are facing a huge challenge.</li> </ul> Maybe modelling is the problem</h2> Something that looks like bad code could be that way because of a failure to properly model the real world.</p> Domain Driven Design (DDD) has much to teach on the matter, and this is a great video on the why modelling could be the problem: Technical debt isn't technical - Einar Høst - DDD Europe 2019 - YouTube</a></p> Small Batches Automated Testing Podcast Episode</h2> Podcast episode Software Delivery in Small Batches: Automated Testing with Jason Swett</a> talks about practical approaches to legacy code, including the idea that you actually can't always avoid writing some more untested code because of the cost-benefit trade-off.</p> "You can’t just slap a test on any change that you make because the underlying functionality might be really complex"</p> </blockquote> The episode describes the evolution of our industry in learning the importance of test coverage for avoiding known risks in software development, and declares that you cannot truly call yourself a professional developer if you are not writing tests.</p> "I always recommend starting with the easiest most trivial stuff, adding some tests in those areas and then working up to the bigger areas"</p> </blockquote> Some relevant thoughts from the episode:</p> Good test coverage is a pre-requisite to any higher-level automation such as continuous-deployment. (1:35)</li> To add tests to a project that has no coverage you have to retrofit test tooling and infrastructure (CI, data management etc). "That can be very non-trivial, even for someone very experienced, ... it's going to be a huge challenge to add that stuff retroactively." (3:10)</li> With regards to the idea of "Declaring that we are going to write tests for every PR from now on." (7:10) "You can't just slap a test on any change that you make because the underlying functionality might be really complex. In order to write a test you might have to have certain setup data to create the state required for that test might be extremely complicated. To expect somebody to include a test with every PR is not realistic. For some features your change might be 10 minutes of work but to get the test setup, that might be weeks of work in order to take the 20-minutes to write the test."</li> "I always recommend starting with the easiest most trivial stuff, adding some tests in those areas and then working up to the bigger areas. It seems that you should start with the most important stuff but that's not realistic because the most important stuff is also the hardest to test."</li> </ul> </li> A useful step might be to just get tests in place on your own machine without worrying about CI initially (they were talking about training a developer, but it's a useful thought for being efficient tackling legacy projects) (9:30)</li> "Having high quality code is almost impossible without good test coverage. Test coverage enables refactoring. If you can't refactor your code it's really hard to keep it in good shape." (17:15)</li> As you flush out bugs by adding more and more coverage: "The type of issues you find over time, the complexity goes up, because they're not really related to the interplay of something known, they're usually related to more and more complications; instead of just one unit of code interacting... [it's] 5 or 10 or 200; these larger components and permutations that you never considered in the first place, because some customer hit it for the first time." Moments where you go "woah I never even knew that was possible" (25:50)</li> "It all comes down to managing risk, risk of the software in question being correct" (30:20) "There's a risk of producing incorrect software, but there's also a risk to the business that the engineers will not be able to keep pace with the business if they have to spend so much extra time just verifying the software, there's a mistake in thinking that writing tests takes more time, the reality is that if you have automated tests then you will be able to work much faster than if you didn't"</li> </ul> </li> On a project that had tests once - "...at some point tests had stopped passing", should you "blow them away or fix them?". (32:22) For an "application that's been developed for two years with no tests with this big team and they don't have the skills; that's going to be a big up front investment, it's going to be a year or two before [adding test coverage] starts to pay off."</li> "... at some point tests had stopped passing and they were never updated - okay do I trust these or not? If they're not passing, do I blow them away or fix them." - In this case blew them away and spent three months writing tests such that he could build new features. "Even now [the project has] low test coverage. This has been a multi-year effort."</li> "This is one of the real problems when it comes to testing, is that if for some reason the people who start the project don't do testing, and the project is successful enough to go on for X number of years without it, then the effort required to add testing after the fact is an order of magnitude than it would have been at the beginning." ... "the can gets kicked down the road."</li> "Frustrated by the attitude of 'let's not spend time making this code high quality because what if this startup fails' but what if this startup succeeds? Yes it may be possible you only survive by cutting corners but not convinced, think it might be a delusion." "The only way to go fast is to go well." ~ Bob Martin</li> </ul> </li> </ul> </li> "Why is good code good? It's fast to work with. I prefer to say 'the code is understandable' because that makes it more clear. Good/bad sounds like inward looking practicing of a craft. 'Not Understandable' is more tangible - that means it's time-consuming and expensive to work with. Understandable code is faster and less expensive to work with." (35:20)</li> </ul> "It all comes down to managing risk"</p> </blockquote> The people on the episode are Ruby developers, but the lessons they talk of apply regardless of language.</p> (Some of the quotations above are paraphrased)</p> Thanks, now I'm even less sure what to do</h2> Are you wrestling with something you don't like in a codebase you have to deal with? I'm guessing that's why you're here. Or maybe someone sent you this along with a rant about the tech debt in the codebase you own.</p> Either way, I suggest slowing down a bit, sitting down together (while maintaining social distancing), and considering how all the things that bother you about the code in front of you fit in to the various taxonomies detailed above. Then use that assessment to make a calm and rational plan about roughly where you want to get to. Then decide on the one</em> next thing you will do towards that. Use the Ron Jeffries approach to get you there without wasting time on things that don't really matter.</p> Practically, that might mean if you have a user story, ticket or Trello card for a feature, it might take longer to do as you include the work to "pay down" some of that badness along the way, knowing that it will improve your overall velocity over time. Be wary of pulling out separate "debt" stories to do, though that can work if the team dynamic is right.</p> To keep your stress levels down make it a shared team problem, have a bit of a laugh about it, take regular breaks in the great outdoors, and support each other.</p> Good luck!</p> I'll leave you with this little song I found on YouTube about bugs:</p> </iframe> Further Reading</h2> If you want to avoid the task at hand by reading the whole internet in the hope that it will help, start here:</p> A lengthy set of opinions on what to do with a huge 10 year old untested pile of spaghetti code: https://softwareengineering.stackexchange.com/questions/416242/is-it-the-correct-practice-to-keep-more-than-10-years-old-spaghetti-legacy-code</a></li> Pluralsight have written an excellent long-form article on the subject: https://www.pluralsight.com/blog/software-development/erasing-tech-debt</a></li> The Engineering Manager on technical debt: https://www.theengineeringmanager.com/growth/how-to-argue-the-space-to-tackle-technical-debt/</a></li> https://adevait.com/software/technical-debt</a></li> https://medium.com/@adamberlinskyschine/wtf-is-technical-debt-b9e9d5f89d9</a></li> https://wtfismyengineertalkingabout.com/2017/03/18/wtf-is-technical-debt/</a></li> https://codingsans.com/blog/technical-debt</a></li> https://builtin.com/software-engineering-perspectives/technical-debt</a></li> https://www.bmc.com/blogs/technical-debt-explained-the-complete-guide-to-understanding-and-dealing-with-technical-debt/</a></li> https://dzone.com/articles/what-technical-debt-it-and-how-to-calculate-it</a></li> https://bigrivertech.com/technical_debt_assessment/</a></li> https://www.cutter.com/consulting/technical-debt-assessment-and-valuation</a></li> https://martinfowler.com/tags/technical%20debt.html</a></li> https://codeclimate.com/blog/10-point-technical-debt-assessment/</a></li> http://thinkapps.com/blog/development/technical-debt-calculation/</a></li> https://www.productplan.com/glossary/technical-debt/</a></li> https://leadership.garden/tips-on-prioritizing-tech-debt</a></li> https://www.steveonstuff.com/2022/01/27/no-such-thing-as-clean-code</a></li> https://leadership.garden/tips-on-prioritizing-tech-debt/</a></li> Youtube: "Types Of Technical Debt And How To Manage Them" ~ Dave Farley</a> "Sense of design" over life of system, not just good design up front.</li> Measure of quality is ability to make changes</li> Metaphors for different types of tech debt: Family loan</li> Bank loan (lack of tests)</li> Loan shark (visit from knuckles)</li> Organised crime loan (fatal if not repaid. Legacy technology)</li> </ul> </li> </ul> </li> Youtube: "7 Ways to Get Ahead of Technical Debt & Keep It Under Control" ~ Rob Walling & Derek Reimer</a> Seen a business acquisition fail due to tech debt found during "due diligence"!</li> </ul> </li> The "Lava Flow" anti-pattern where code is written, ossifies and leaves an archaeological record of all previous design attempts, libraries and technologies in the current production code rather than the git logs where they belong https://mikehadlow.blogspot.com/2014/12/the-lava-layer-anti-pattern.html</a></li> http://antipatterns.com/lavaflow.htm</a></li> https://en.wikipedia.org/wiki/Lava_flow_(programming)</a></li> </ul> </li> </ul> Personal backlogs 2020-06-30T00:00:00+00:00 What's wrong with our current backlog / icebucket / todo list?</h2> Do you have a backlog of doom? A bottomless bucket full of probably unimportant things you possibly should do, interspersed with the occasional really import thing that will bite you hard if you don't do it? Do you spend time repeatedly refining the list to make sure you don't miss the important things? Or do you just ignore it and hope luck will save you from the things you let slip?</p> I've known for a long time that basecamp promote having no backlog at all</a>, based on the idea that it takes too much time and anything important will come up again anyway. To some extent I agree, but I have also found serious issues in systems that are perhaps less visible to the naked eye which will for sure have a consequence if not addressed (maybe something that is quietly corrupting data or failing to store it, or maybe a security issue).</p> So what should we do instead?</h2> The answer came to me when I read basecamp's new Shape Up book (summarized here)</a> that describes their unique way of working. They don't have a shared backlog, instead they leave it to each person in the organisation to use whatever method they choose to remember what they</em> think is important, and then bring that to the table for consideration at the scheduled times that new work is considered.</p> The full shape-up process is a bit too much of a jump for the teams I usually work on, however I shared the above with a team I worked with, and they took it and came up with an excellent best-of-both-worlds solution. And it's so good I think everyone should use it.</p> This solution...</p> perfectly balances the elimination of a backlog-of-doom nightmare,</li> addresses the need to not lose important ideas as team members change,</li> and adds the ability to collaborate.</li> is ridiculously simple</li> </ul> Trello</h3> Set it up as follows: Set up a new trello board, name it "shared backlog" or similar, create a list (column) for every team member with their name (and optionally role) as the list title.</p> </p> Now when anyone thinks of anything that the team should possibly do, they add a card to their own list</strong> on the shared backlog trello board. It is now clear who owns the idea and is responsible for campaigning to get it prioritized.</p> In preparation for the next planning session each person organises their own list so that the things they think are most important are on the top.</p> On a regular basis, (say fortnightly during a sprint planning session or similar), the team goes through the board, and each person talks about their top one or two cards to say why they think it should be included in the sprint. The cards that are chosen by the team get added to the sprint to be done as part of the normal work for the team, thus making sure they actually happen.</p> Simple eh?</p> Here's an example board https://trello.com/b/cxRXPgcb/personal-backlogs-example</a>, you can use it as a template if you wish.</p> Jira</h3> If you're using Jira you can achieve the same effect by abusing the "sprint" feature. Just create some extra sprints with the "Create spring" button and instead of calling them "sprint 25" etc, call them "Tech backlog", "Design Backlog", "Customer Service Backlog" etc. Then you can nicely group each role's needs and they can manage their own priorities within their own backlog. The actual "backlog" in Jira can now be just for true user stories / product features.</p> </p> Useful?</h3> Let me know</a> if you use this approach and if it works for you or not.</p> What about normal feature development?</h2> You could theoretically have the product owner manage all the forthcoming product features in their column. I think it is better for them to continue to manage the product's feature backlog in the existing jira/trello/github project system, which will now be beautifully free from tech debt, random ideas and other clutter. This is what we did and it worked great.</p> Bonus ideas</h2> Add a label "next sprint", and get people to tag the cards they think should happen next.</li> Technical debt often ends up here, read this http://laughingmeme.org/2016/01/10/towards-an-understanding-of-technical-debt/</a> to be able to talk more coherently about this topic.</li> Turn on voting in trello and vote on each other's cards.</li> </ul> Did it work for you?</h2> This setup worked superbly for us as a team and I'd recommend it to anyone who'll listen.</p> We no longer have the overhead of a product owner losing the will to live trying to understand cryptic tech debt cards left by a previous developer, wondering if they were trivial or pending catastrophes.</p> When someone leaves, the new person or others on the team can look over their cards and decide what to move into their own column.</p> Also available on the podcast</h2> If you like this, you might want to have a listen to me and David talking about this and other lessons learned in episode 8 of software should be free - Hiding from Covid-19 and the end of 2 years at DfE</a>.</p> Further reading</h2> A very similar idea of "JIT selection from independent streams" by Jason Cohen (aka A Smart Bear)</a> which is well worth reading.</li> </ul> Fast backup to external drive with lz4 2020-04-23T00:00:00+00:00 The simplest backup is to just tar-gz your home directory, but it's painfully slow. A friend</a> put me on to lz4</a> which isn't installed by default in ubuntu & mint, it moves the bottleneck from the compressor to disk i/o for my spinning rust usb disks.</p> The default output of this is basically silence for two hours while it runs which isn't great. Enter pv</code> (pipe viewer</a>) which can show you progress based on byte count.</p> Install lz4 and pv with:</p> sudo apt install liblz4-tool pv</span></span></code></pre> And then pipe the output of tar through pv and into lz4</p> Here's a quick script for backing up my home folder to an external drive; customize to suit your needs:</p> #!</span>/bin/sh</span></span> #</span> https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh</span></span> echo</span> "</span>Opening/creating backup folder...</span>"</span></span> mountpoint</span>=</span>/</span>m</span>e</span>d</span>i</span>a</span>/</span>t</span>i</span>m</span>/</span>b</span>a</span>c</span>k</span>u</span>p</span>/</span></span> cd</span> "</span>$</span>mountpoint</span>"</span></span> mkdir</span> -</span>p</span> fox</span></span> cd</span> fox</span></span> base</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span></span> src</span>=</span>t</span>i</span>m</span></span> echo</span> "</span>Getting source folder size...</span>"</span></span> size_bytes</span>=</span>`</span>du</span> -</span>sb</span> "</span>$</span>base</span>/</span>$</span>src</span>"</span> \|</span> awk</span> '</span>{print $1}</span>'</span>`</span></span> echo</span> "</span>$</span>size_bytes</span> bytes to backup.</span>"</span></span> echo</span> "</span>Backing up...</span>"</span></span> tar</span> -</span>cpC</span> $</span>base</span> $</span>src</span> -</span>P</span> \|</span> pv</span> -</span>s</span> "</span>$</span>size_bytes</span>"</span> \|</span> lz4</span> >></span> "</span>$(</span>date</span> -</span>d</span> "</span>today</span>"</span> +</span>"</span>%Y%m%d-%H%M</span>"</span>)</span>-home.tar.lz4</span>"</span></span> echo</span> "</span>Done.</span>"</span></span></code></pre> This script is also available here: https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh</a></em></p> Things to watch out for</h2> Progress is inaccurate and variable because it's based on input bytes processed vs total, but the speed is limited by output bytes to the spinning rust backup disk, and the ratio varies with the compressability of the input data.</li> Finding a file will require reading or decompressing the entire archive because of the way tar works.</li> </ul> Keeping it simple</h2> There's lots of great backup tools for backing up home but most of them create obscure custom formatted backups, often with incremental backups and chunked up files for efficient use of space. While that's great I know from experience that you don't always have all the right tools available to unpack such things when you really need them, and there's always the chance that you are missing an important incremental. In short complexity is worrying when it comes to backup (and especially restore).</p> References:</h2> https://stackoverflow.com/questions/24063846/how-to-use-tar-with-lz4#24086155</a></li> https://superuser.com/questions/168749/is-there-a-way-to-see-any-tar-progress-per-file/665181#665181</a></li> https://stackoverflow.com/questions/8228047/adding-timestamp-to-a-filename-with-mv-in-bash/8229220#8229220</a></li> </ul> Question: where are we at with dynamic static sites? 2020-04-14T00:00:00+00:00 So tell me about dynamic static sites...</p> History lesson</h1> I've been a software engineer primarily creating dynamic websites for a loooong time. First there was a server with html files and caching proxies. Then there were scripting extensions (cgi, php, asp-classic etc) to spice up your html, then there were full mature web frameworks for all the things (asp.net mvc, ruby on rails etc).</p> But then we got traffic and libraries and platforms and sdks and databases in the cloud and everything became slow. 😞</p> So we added CDNs to make it all fast, and cache invalidation to make the engineers cry.</p> Now what's old is new again, but with a twist, static sites pre-generated by dynamic code (such as my blog created with jekyll). We get the benefits of old</p> CDNs love static sites; caching and proxying work again; and cache invalidation still makes us cry.</li> </ul> The new saviour is serverless (functions as a service) so that we can still add dynamic stuff to these static sites.</p> Yes I can go read endless articles about how to do</em> all this stuff, and turn my CV and skills on their head so I can sell the hot new thing. But what I actually want to know now is....</p> What are you</em> doing with dynamic static sites?</h1> Have you tried it? As a pet project? As a commercial project?</p> Did it go well? Is it the new future or is it just for niche needs?</p> I feel like because this is all sooo</em> new, that it's not clear yet how well adopted this stuff is or will be, and how much of the old is being thrown out and rewritten.</p> My opinion...</h1> ...based on good knowledge of tech but not based on any actual experience is that it depends entirely on the type of thing you are building:</p> Low volume intranet/extranet style sites - don't bother, asp.net / rails is fast enough, and it will cost you more to develop a blended static/dynamic thing.</li> E-commerce: totally worth it because you can handle any spike in traffic to your front page, and serverless lets you seamlessly scale the dynamic parts. Caveat it will be more expensive to build</strong> but that's okay because money in is proportional to how many customers you can serve well.</li> Government or other highly accessible services - don't do it. You can't meet the accessibility needs if half your site's critical behaviour is done in client-side javascript with calls off to serverless functions in the cloud. I might be wrong about this, I hope I am. Maybe it's possible but just hard.</li> Your blog and other traditionally CMS driven sites: just use jekyll for your own blog</a> and put it on github pages. For more complicated commercial sites static site generation is a big win in terms of technical complexity reduction, reliability improvements and speed improvements for users. Which static site generator you use will depend on your organisation's internal needs. (Can everyone use git? Or do you need a UI for people to type in? How much money to you have?)</li> Etc. (there are sooo many reasons for tech, I've just picked a few)</li> </ul> Speak out now</h1> Before you go, comment below.</p> What have your experiences been with the shift to dynamic-static sites? Have you done any of it? Do you think I'm right/wrong about any of this?</p> What you should do between contracts 2020-04-09T00:00:00+00:00 Strange times with Covid-19</h1> I can't post this without mentioning the context I'm writing in of the coronavirus pandemic lock-down.</p> I consider myself extremely lucky to be in the line of work I am that allows me the flexibility to continue working remotely whilst still protecting myself and those around me by avoiding contact.</p> What I normally do</h2> In calmer times I'd have taken the opportunity to spend at least a couple of weeks with family (have small people and time with them is preciously fleeting). Then get my life in order, then go hell for leather getting into the next thing I can help build.</p> Some of my fellow contractors like to make sure they have the next contract lined up ready for the finish date of their current contract. While this is optimal for revenue I don't do this because:</p> I find it is a full time job generating, tracking and dealing with contracting leads</li> I wouldn't want to be distracted from the current client's work to the next thing</li> I wouldn't want to half-arse the contract hunt and not get the optimal contract for myself and the next client.</li> There is a tendency for last-minute contract extensions to appear which would mean letting down one or other client.</li> </ul> So after a contract is completely finished, and some time with family, only then do I take the contract hunt seriously.</p> What I'm doing this time</h2> Given the uncertainty caused by COVID-19 + IR35 (by the way COVID is short for Corona Virus Disease) and the fact I just moved house (i.e. less cash reserves than usual) I can't take it too easy this time. Even though the IR35 changes have been delayed a year a lot of the damage has been done so the contractor market is challenging at the moment.</p> I finished my contract with DfE on a Tuesday, allowed myself till the following Monday for uninterrupted family time. (Okay almost uninterrupted, I can't really put technology down for that long and I also have my charitable work for DogLost</a> that takes up some time.)</p> Now that is over, I'm doing roughly 9am-1pm on the business related tasks suggested below.</p> What should you do to make the most of finishing a contract?</h1> Step 1. Take a break</h2> Contracting is in my experience much more intense than permanent employment. This is in fact how I like it. But in order to be able to give your all to the next contract I think it is important to give yourself the space to recharge.</p> When in a contract it's easy to end up with tunnel vision, especially for one as long as my last one (2 years!), where all you can think about is how it was done there and how you reacted to the influences around you during that time. One of the strengths you can bring to a contract as a contractor is the broader perspective from seeing many organisations, approaches, technologies and people. It requires a bit of down-time for your latest experience to sink in once you are out of the hustle and bustle of delivery, and for it to be merged into your bank of knowledge.</p> What better way to end a contract than with good chunk of time with family and friends while your subconsious churns through and processes everything that's happened over the last contract.</p> I believe if you dive headlong into the next thing with nothing more than a normal weekend off then your brain will not have the opportunity to properly process what you've learnt before being overwhelmed with an influx of new information from a new client.</p> Step 2. Update your online presence and CV</h2> You've probably learned a lot and changed a bit since you started the last contract.</p> Review and update all your profiles:</p> LinkedIn</li> Twitter (make a new pinned tweet!)</li> GitHub</li> StackOverflow</li> etc.</li> </ul> Think about what your next client will be interested in and what they want help with and make sure your profiles provide evidence that you have done similar things before and can get results.</p> Step 3. Do some writing</h2> The best way to shape your thoughts on the whole thing is to put them into writing. So spend some time blogging like I'm doing now, and make sure to cross-post your articles to the places where your customers will see them. You should own your own content, keep it on your own domain on your own blog where no platform can take away your audience, then cross-post to places like dev.to</a>, LinkedIn and Medium (if that's your thing) with links back to your own domain. Better still on your posts ask people to sign up to your mailing list and email them when you post new articles etc.</p> Step 4. Catch up with old acquaintances</h2> It's hard when you're flat out with life and contracts to keep in touch with everyone. Reach out to old business and personal friends. You never know it might kick off your next opportunity, or help you learn something about yourself that's useful.</p> Step 5. Go all guns blazing on getting the next piece of work</h2> "You will get all you want in life if you help enough other people get what they want," ~ Zig Ziglar</p> </blockquote> Figure out who'd value your skills most even in these difficult times and have at it.</p> You might find my previous article "How to find contract developer jobs"</a> useful at this point.</p> Personally I'm looking to also increase the value I provide beyond implementation work. This might take the form of team lead roles, or some digital transformation consultancy.</p> ~ End ~</h1> What do you do between contracts or how do you avoid having gaps? Are you between contracts now thanks to IR35?</p> Time to "Shape Up" your SCRUM processes? The new thing from Basecamp 2019-11-26T00:00:00+00:00 Basecamp, who are famous for carving out their own path in the software world have documented and shared their own way of defining and building their software-as-a-serice (SaaS) product, also known as "basecamp". You can read the whole thing here: https://basecamp.com/shapeup</a>.</p> If like me you've mostly worked with SCRUM/Agile/Kanban style teams you should pay attention to the ideas they share. Unlike many people who teach and follow the wikipedia truth of SCRUM as the</em> way of building software, basecamp have not satisfied themselves with cargo-culting "industry best practice" and have instead carefully honed their own particular way of getting the best out of their resources.</p> Although the "ShapeUp" process is finely tuned to suit basecamp, there are many things in there that the average SCRUM team could take inspiration and improvement from without throwing out everything in place at the moment. An example I particularly like is the way everyone can bring a pitch to the betting table, from their own personal backlog of priorities, contrasted with the usual approach of throwing yet another another card/story onto the team-shared dumpster-fire of a backlog / todo-list and hoping it somehow magically gets to the top of the list just before all your chickens come home to roost.</p> Key features of shaping up</h1> Shaping</h2> An iterative process of taking concepts from initial idea to something that could be built with bounded risk and expectations.</li> A largely solo or pair exercise, but done in consultation with experts on the wider team.</li> </ul> Pitch</h2> The output of shaping.</li> A detailed but high-level document outlining what should and shouldn't be built, with room for discretion and design work by the designers and programmers.</li> </ul> Bets</h2> There is a meeting where pitches are discussed, prioritised and scheduled for the next cycle of work, they "take a bet" on a pitch.</li> If a pitch blows its timebox, then the default is always to stop and it doesn't get any special priority in the next cycle. This limits the risk of runaway unstoppable projects.</li> </ul> 6-week cycle with gaps</h2> Basecamp operate a 6-week period of building what was decided in shaping and betting. This can either be one 6 week bet or set of 2 week bets.</p> Once this is done there is a 2-week gap where people can do maintenance, tech debt paydown, plan new things etc.</p> If you'd like to hear a bit more about ShapeUp, have a listen to https://pod.timwise.co.uk/5</a> where I go over their process in a bit more detail with regular co-host David.</p> I've also written up a nice approach to avoiding a "backlog of doom" in my personal backlogs post</a>.</p> Should you rebase or merge to update feature branches in git? 2019-10-14T00:00:00+00:00 You have a "feature branch" in git that you've been working on for a while but now main</code> or master</code> has moved on. You know of merge</code> and rebase</code>, but which one should you use? And what can you do to avoid being in this position in the first-place?</p> TLDR</h2> Try rebase. If that dissolves into conflict-resolution-hell then give up, merge master into your branch and move on.</p> The Options</h2> You need to bring your feature branch up to date with with master to flush out any incompatibilities and deal with any merge conflicts.</p> You have two common choices:</p> Merge</strong> origin/master</code> into your branch.</li> Rebase</strong> your branch onto origin/master</code> and force-push.</li> </ul> The Trade-offs</h2> A blanket rule here either for merge or rebase is unhelpful because there are trade-offs</strong> to be made that vary depending on the specific circumstances. (Isn't that always the answer with git?! "It depends!")</p> Should You Merge?</h3> A merge from master</code> is done like this:</p> git fetch</span></span> git merge origin/master</span></span> git push</span></span></code></pre>Merge - The Good</h4> 👍 Reliable no-brainer that anyone can follow by rote.</li> 👍 Resolve conflicts only once.</li> 👍 Accurate representation of what happened over time.</li> 👍 Avoids retrospectively introducing bugs and test failures into commits that used to be valid</a>.</li> 👍 Avoids re-writing previously shared branch, which can confuse less experienced git users if they are working with you on the branch.</li> </ul> Merge - The Bad</h4> 👎 Doing this repeatedly makes for a messy history for little or no benefit.</li> 👎 Significant merges from master makes it harder/impossible to then go back and clean your branch's commits with a git rebase --interactive</code>.</li> 👎 Tends to generate wide tramlines in the commit history</a> that can be very hard to follow when looking back to find out when/why something was done. (mitigated by git log --first-parent</code>, until you need to dig into a branch).</li> </ul> Should You Rebase?</h3> A rebase onto master is done like this:</p> git fetch</span></span> git rebase origin/master</span></span> git push --force-with-lease</span></span></code></pre>Rebase - The Good</h4> 👍 Avoids tramlines generated by long-lived feature branches branch</li> 👍 Makes resultant history in master</code> much easier to follow</li> 👍 Reflects the intention more clearly of "merge these commits into master" as opposed to "here's how I flailed my way to a working thing"</li> </ul> Rebase - The Bad</h4> 👎 Can confuse less experienced git users if they are working with you on the branch (the answer is usually for them to run git pull --rebase</code>)</li> 👎 Results in resolving conflicts multiple times (screencast)</a></li> 👎 Loses chronological order of creation of code (personally I think this is less important than a series of clean intentional patches to be applied to the codebase when merged to master</code>)</li> 👎 Could in somewhat rare circumstances retrospectively introduce bugs and test failures into commits that used to be valid</a></li> </ul> Heuristics To Use</h2> Try rebase. If that dissolves into conflict-resolution-hell then give up, merge master into your branch and move on.</p> "Try rebase. If that dissolves into conflict-resolution-hell then give up, merge master into your branch and move on."</p> ~ Tim Abell</p> Tweet this</a></p> </blockquote> Rebase is my preferred approach until:</p> Rebase becomes too costly to fix up due to conflicts with master</code>, or</li> I become aware of an incompatibility with master</code> that changes the meaning of the previous commits and needs serious work to resolve.</li> </ul> You can</em> usually make a difficult rebase work, and I've hunkered down and tackled probably more than I should have in the name of perfect history graphs.</p> The problem with a tricky rebase is that if you are doing this for business and not just for fun then there is a major time cost for only a marginal benefit.</p> How to make it through rebase conflicts unscathed</h2> If you decide to battle on with rebase in-spite of conflicts then my tip for you is:</p> Don't jump straight to the "correct" code when fixing each commit's conflict, as that guarantees the next commit won't apply.</p> Instead as you work through the rebase make each commit apply with its original meaning and nothing more.</p> It's worth remembering that each commit on your branch describes how to change the source code from a before-state to an after-state; so if you change the after-state of one patch, then the next patch will no longer apply.</p> How to avoid the pain of rebases and merges entirely</h2> Pain around this topic is likely a symptom of not breaking down your stories / pull requests / features into small enough chunks. On a fast moving team master</code> is very fluid and any large & long-running branches will be hard to review and merge. Try to chip off smaller increments and ship those, maybe using feature flags or hidden features (those with no visible way of getting to them).</p> More Resources</h2> In general merge vs rebase generates much debate, such as that found on stackoverflow: https://stackoverflow.com/questions/804115/when-do-you-use-git-rebase-instead-of-git-merge</a> but it is often lacking context.</p> There are many other articles on the merge/rebase topic such as https://derekgourlay.com/blog/git-when-to-merge-vs-when-to-rebase/</a> but I couldn't see anything that matched my heuristic for tackling feature branch updates so I wrote this one.</p> "Semantic merge conflicts</a>" are where git reports no conflict but nonetheless the code is broken.</p> I also wrote "GitHub rebase and squash considered harmful</a>" which address a github specific horror.</p> Get in touch</h2> Hey there! Thanks for reading!</p> This post gets far more traffic than anything else on my blog. I'd love to know what brought you here and if the above was helpful.</p> Please take a moment to fire an email to me at tim@timwise.co.uk</a> and tell me a bit about yourself.</p> Why I made SQL Schema Explorer open source 2019-10-08T00:00:00+00:00 What is SQL Schema Explorer?</h1> It's a database schema & data browser that can follow foreign keys and draw diagrams that I spent years building because I wanted it to exist and I thought it might make a good first product.</p> Why charge?</h1> I'm pretty busy these days but I like working on software and building things like this, to justify the amount of time that goes in I can no longer just have it as a time-consuming hobby. I've also been learning about entrepreneurship as a possible extension or next step from time-for-money contract-programming services.</p> The idea was that if I could get enough yearly subscribers to the software then I could get balsamiq-like success and focus on making it better and better for the people using it.</p> Why open source now?</h1> I got plenty of interest and interactions by my standards, more than I've ever had for something I've published; but I never got more than one good friend paid up beyond a generous trial. The conversations I had with people were full of enthusiasm for the idea but not for parting with money to buy it. One particular conversation was very positive until the idea of putting the theoretical purchase in front of the manager was raised and then suddenly it didn't seem viable at all; which I think speaks volumes. Useful, but not useful enough to risk any political capital to get it, even for a pretty insignificant amount of money. I then had a coaching call with Justin Jackson</a> and as part of that he suggested putting it to one side. Up to this point I'd been unable to leave the idea alone; it had been something I really wanted to exist. I've always worked with relational databases, and this seemed like a painful gap. At this point I had finally added every feature that I'd ever dreamed it should have, and the intrinsic motivation to just build for the sake of building was waning, and without the demands from paying users for more features it's hard to justify continuing to pour hours in even if it is on the train.</p> At this point there were 100 people on the mailing list (drip sent my first bill!), one paying user/fan (Hello David! Many thanks!), and zero MRR (monthly recurring revenue). I'd spent a bit of money on hosting (I'm still paying digital ocean to keep the demo site</a> up).</p> I didn't want to let down the few people who had really liked it by having it just vanish into the ether like so many failed startups. Fortunately this is downloadable and installable software, not a SaaS that would have to be shut down. I've always liked the open-source ethos, and given this was no longer a direct financial opportunity it made sense to me to give it away freely under a copyleft license.</p> It feels like having this as a portfolio piece given I sell coding for a living won't do me any harm, and it makes it easier to take with me to client projects, where I can use it to help them improve their database.</p> Schema Explorer for client work</h1> Here's some things I've been using schema explorer for to help my contracting clients</p> Finding missing foreign keys without looking up complex pg_schema</code> queries</li> Analysing data in tables and sharing screenshots with the teams</li> Sharing screenshots of the generated diagrams to improve team understanding</li> Using it as a tool for discussing database design</li> Finding records needed for troubleshooting and debugging</li> Improving my mental model of the database structure</li> </ul> Tell me more</h1> It's now A-GPL</a> which requires sharing any modified source code even if you are just letting users access the generated site. If I'm giving my years of hard work for free, I think it's only fair to require people to contribute their improvements back to the community.</p> You can find the source code at https://github.com/timabell/schema-explorer</a></p> Feedback</h1> Please do get in touch if this sparked a thought or interest. Just a simple email will do tim@timwise.co.uk</a></p> Thanks for listening.</p> Laptop setup notes 2019-08-20T00:00:00+00:00 This is mostly for my benefit YMMV. Used for XPS13 and XPS15. Last run with Linux Mint Cinnamon 22</p> Things to go from blank machine to fully functioning work laptop.</p> Why not use machine images?</h2> Because when you want to upgrade your OS image you have to do this anyway.</p> But this will break!</h2> Indeed, and it does, a continuous labour of love. Ideas welcome!</p> OS install</h2> Download mint cinnamon x64 torrent</a></li> Burn to USB stick</li> Configure Dell XPS 13 to use AHCI mode</a> (one time fix)</li> Boot to stick with F12</code></li> Install mint Full disk encryption (LUKS), default whole disk partitioning</li> </ul> </li> Run updates and reboot sudo apt update && sudo apt upgrade</code></li> </ul> Hibernate / hybrid sleep (unfinished)</h2> Not working at moment, needs more research.</p> Enable hibernate? Bigger swap?</li> </ul> </li> </ul> Unfinished research into missing hibernation</h3> https://superuser.com/questions/1539378/what-size-swap-partition-will-allow-a-mint-19-3-installation-with-64g-ram-to-hib?noredirect=1&lq=1</a></li> https://superuser.com/questions/1434301/enabling-hibernate-on-linux-mint-19-1?noredirect=1&lq=1</a></li> https://help.ubuntu.com/community/SwapFaq</a></li> https://unix.stackexchange.com/questions/568093/enable-hibernation-in-power-management</a></li> https://dewaka.com/blog/2021/04/08/linux-mint-hibernate/</a></li> https://www.fosslinux.com/45454/enable-hibernate-mode-linux-mint.htm</a></li> https://askubuntu.com/questions/12383/how-to-go-automatically-from-suspend-into-hibernate</a></li> https://www.reddit.com/r/linuxmint/comments/93ta9u/enable_hibernation_in_linux_mint_19_tara/</a></li> </ul> Packaged software install</h2> Bootstrap my dotmatrix package list my bootstrap file</a> like this (don't laugh/cringe):</li> </ul> curl https://raw.githubusercontent.com/timabell/dotmatrix/main/software/bootstrap.sh \| sh</span></span></code></pre> This installs my usual package list</a></p> Hardware</h2> Video</h3> Disable nvidia GPU in favour of intel card</li> </ul> sudo prime-select intel</span></span></code></pre>Bluetooth</h3> Enable fast-connect etc for bluetooth keyboard/mouse</a>.</p> zsh</h2> Switch to z-shell chsh -s /usr/bin/zsh</code></li> </ul> Ref: https://askubuntu.com/questions/131823/how-to-make-zsh-the-default-shell/131838#131838</a></p> inotify</h2> inotify increase for: RubyMine</li> Guard - https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers</a></li> Syncthing - https://docs.syncthing.net/users/faq.html#inotify-limits</a></li> </ul> </li> </ul> echo fs.inotify.max_user_watches=524288 \| sudo tee -a /etc/sysctl.conf && sudo sysctl -p</span></span></code></pre>File sync with syncthing</h2> Run syncthing-gtk from system menu (auto-installs syncthing binary) Configure syncthing</a> not to route over internet (no relay, no public discovery, no NAT traversal)</li> </ul> </li> Configure firewall-config</a> to allow syncthing Add manual service syncthing</li> Allow tcp 22000 and 21027 udp</li> Make home network use home zone</li> Add syncthing to home zone</li> Runtime to permanent</li> </ul> </li> </ul> </p> Remove old install machine name from syncthing network</li> Tell new and old devices about each other in syncthing</li> watch ~/Documents</code> etc. magically sync (amazing)</li> </ul> All the things from dotmatrix</h2> cd repo/dotmatrix/</span></span> bin/symlinks.sh</span></span> bin/install</span></span> software/delta.sh</span></span> # etc</span></span></code></pre>Link local git config</h2> cd ~</span></span> ln -s ~/Documents/config/.gitconfig.local</span></span></code></pre>Desktop/system config</h2> Setup touchpad Set mouse and trackpad speed/acceleration to max</li> Enable horizontal touchpad scroll</li> </ul> </li> </ul> </p> Setup terminal colours and turn off transparency</li> </ul> </p> Turn off all the system sounds and turn the volume down to 0%.</li> Customize time in task bar to 📅 %a %e %b %Y 🕓 %H:%M:%S</code>.</li> Lock screen with Win+L</code> - start > keyboard > system > lock screen (Windows user habits).</li> Turn on automatic updates in update manager preferences, and automatic obsolete kernal removal.</li> Configure nemo file manager to always use list view.</li> Add workspace switcher applet to taskbar.</li> </ul> Firefox</h2> Sign-in to sync</p> </li> Remove firefox tabs (because I use Tree Style Tabs</a>)</p> cd ~/.mozilla/firefox/<profile_id_folder>/</code></li> mkdir chrome</code></li> echo '#TabsToolbar { visibility: collapse !important; }' >> chrome/userChrome.css</code></li> about:config > toolkit.legacyUserProfileCustomizations.stylesheets</code> > true</code></li> restart firefox</li> </ul> </li> </ul> Reference: https://superuser.com/questions/1268732/how-to-hide-tab-bar-tabstrip-in-firefox-57-quantum#1268734</a></p> Startup list</h2> Menu > Startup Applications</li> Adjust to current tastes</li> Add additional startup apps: Syncthing GTK</li> </ul> </li> </ul> Non-apt program installations</h2> Use Applets config to download and add Pomodoro timer</a> in task bar.</li> Install jetbrains toolbox</a></li> Install flatpak apps from dotmatrix</a>, sign in to them all Give flatpak access wider home folder access</a> (for sending pics) with flatseal</a></li> </ul> </li> Install all the other things in dotmatrix/software</a></li> </ul> SSH</h2> Generate ssh key ssh-keygen</code></li> Add public key to github</a></li> </ul> Postgres</h2> Use a docker image</a></p> XPS firmware updates</h2> fwupdmgr get-devices</span></span> fwupdmgr get-updates</span></span> fwupdmgr update</span></span></code></pre> refs:</p> https://fwupd.org/lvfs/docs/users</a></li> </ul> Shutter</h2> Change default save location to ~/tmp/shutter</code> (new folder)</li> change filename to shutter_%Y%-m-%d_%NN</code></li> </ul> Thunderbird</h2> Restore recent backup (from data sync)</li> </ul> asdf</h2> install asdf version manager</a> for all the things (ruby, node, golang etc.)</li> </ul> NodeJs setup</h2> asdf plugin-add nodejs</span></span> bash ~/.asdf/plugins/nodejs/bin/import-release-team-keyring</span></span> asdf install nodejs 8.16.1</span></span> node -v</span></span></code></pre> Keyring because nodejs packages are signed and validated https://github.com/asdf-vm/asdf-nodejs#install</a></p> Ruby</h2> asdf plugin add ruby</span></span> cd some-project</span></span> asdf install</span></span> gem install bundler</span></span></code></pre>See also</h2> https://github.com/thoughtbot/laptop</a></li> </ul> Why every team needs a Delivery Manager (DM) 2019-07-08T00:00:00+00:00 I was explaining to someone the team structure I'm working in at the Department for Education's DfE Digital</a> and realised not everyone has seen this structure at work and the magic it brings.</p> Team structures</h2> GDS (government) team structure</h3> In short the team looks something like this:</p> Product Manager</li> Delivery Manager</li> Developer(s)</li> User Researcher(s)</li> Content Designer(s)</li> Interaction Designer(s)</li> </ul> At the top of the whole thing there's a "Service Owner".</p> Involved from the very early stages, but a bit less in the daily grind of building technical solutions, you'll find a "Service Designer".</p> It's a very flat structure. It's a group of peers who all bring their own strengths to the team, but with no hard boundaries, we all muck in with everything now and then. I particularly like helping with user research as it keeps me connected with the users. I'm constantly feeling guilty for not attending enough user research!</p> This structure is modelled on the structure of a team at the Government Digital Service </a>(GDS)</p> Aside: Product who?</h4> You may also not be familiar with Product Manager (I've not seen it as much in private industry), so as an aside let me explain:</p> The Product Manager is more widely known in industry as Product Owner</del> (well that's what I thought, actually it turns out product owner and product manager are not quite the same</a> and there is depth to the product manager/owner question</a>). This is someone who figures out the balance of features in a service and the software that supports it, based on all the various constraints and inputs (e.g. user need, security, stability, internal need). One output of this role is prioritization of work.</p> Private company team structure</h3> The kind of structure I've more commonly see in the private sector is something like this:</p> Project Manager</li> Business Analyst (BA)</li> SCRUM Master (in title if not in deed)</li> Developer(s)</li> Testers(s) / QA (Quality Assurance)</li> </ul> Which is better?</h3> I've seen good and not so good people in all of the roles, and number of roles varies with team size.</p> I think the GDS-style team structure is superior when it comes to shipping user-value as a development team.</p> DM for the win</h2> The role I want to focus on here (though they are all awesome) is Delivery Manager as it's the most uncommon in my experience.</p> I've personally never seen delivery management called out as a specific named role outside of my work for "Digital" teams in UK Government. I hope this role becomes more common everywhere.</p> So what does a Delivery Manager actually do?</p> Greasing the enterprise wheels</h3> In any organisation beyond about 20 people the bureaucracy starts to sneak in, layers of rules are added as a business tries to avoid repeating previous mistakes, and as you hit enterprise scale it's positively stifling, to the point that people issues and an inability to even open Microsoft Word (TM) (R) ((c) 1908)</em> on a work computer without signing 3 forms in blood</strong>, become nearly impossible. The thing is, a dev team is usually building something that hasn't been built before, maybe as part of a major change to how the organisation functions</strong>.</p> Web development is more than just taking your sales brochures and putting them online, if that was all we wanted I'd give you a copy of WordPress and find something else to build.</p> Sometimes you can hide the team away in a skunkworks</a> project, but that can't last forever and eventually the old meets the new, and suddenly the whole project is at risk of being rejected by the legacy organisation. The productivity can be sucked out of a development team for good, they can end up unable to deliver anything.</p> This is where a Delivery Manager</strong> role is golden; they deal with making sure the organisation's goals to have this new service / system / product / software are able to be progressed, conversations & endless meetings had, finance obtained, hardware procured, forms filled, security audits managed sensibly, pile-on demands defended … the list is endless. By having this role the team can focus on building and shipping, comfortable that they won't be sidetracked or blocked.</p> I still like the way the wonderful Mark 'Stanners' Stanley put it back in 2012. I think this cuts to best thing about it:</p> The delivery manager is there to remove any and all things that are hindering or ‘blocking’ them, so the team can deliver the product.</p> ~ Mark Stanley - https://gds.blog.gov.uk/2012/12/12/a-day-in-the-life-of-a-delivery-manager/</a></p> </blockquote> The approach is not new, back in the year 2000 Joel Spolsky wrote:</p> Compare this to Microsoft, where things are done at the lowest level, and most managers act like their most important job is to run around the room, moving the furniture out of the way, so people can concentrate on their work.</p> ~ Joel Spolsky - https://www.joelonsoftware.com/2000/03/19/two-stories/</a></p> </blockquote> Running great delivery teams</h3> You'll also find the Delivery Manager doing inward-looking work similar to the role of "SCRUM Master". This includes setting up the right agile environment (which may SCRUM, Kanban, or something else), tackling any dysfunction and poor team performance that has built up, keeping in touch with all the team to make sure they are happy and productive, running team retrospective sessions, and sometimes hiring-and-firing to get the best team in place.</p> Is it good for devs?</h3> As a developer on an Agile team working inside a government team (not for the first time), I am eternally grateful for all the things I don't</em> have to worry about because we have a DM. It's not that I'm not capable, I'll happily turn my hand to the biggest problem of the day for a team at any time, but the code won't write itself and that's what I was hired to be a pro at. Having a DM lets me focus more creating software that actually does a job for our users, and makes sure we can do what's needed to make great systems even if that reaches beyond the team's original boundaries.</p> Further reading</h2> https://www.gov.uk/service-manual/the-team/what-each-role-does-in-service-team</a></li> https://bizfluent.com/info-8704880-role-software-delivery-manager.html</a></li> http://itsadeliverything.com/delivery-manager-a-new-role-for-an-agile-world</a></li> https://it.toolbox.com/blogs/deliverydr/what-is-delivery-management-070606</a></li> https://www.greatsampleresume.com/job-responsibilities/delivery-manager-responsibilities/</a></li> https://work.chron.com/job-description-service-delivery-manager-24247.html</a></li> https://www.quora.com/Whats-the-difference-between-a-delivery-manager-and-a-project-manager</a></li> https://melissaperri.com/blog/2017/06/29/product-manager-vs-product-owner</a></li> </ul> How to find contract developer clients 2019-06-26T00:00:00+00:00 Handling recruiters</h2> Before you dive in, you might want to look at my page on dealing with recruiters</a>, especially if you are going to be dealing with multiple recruiters at once.</p> Rate research with ITJobsWatch.co.uk</h2> IT JobsWatch</a> is a great guide to market prices. You can see numbers, graphs, trends, compare tech, and see what appears together.</p> It's also a useful source of ideas for considering what might be a good investment in saleable skills.</p> Note the volume as well as the price, for example C# is massively higher volume than Ruby.</p> Lead sources</h2> In rough order of the success I've had with them over the 7 years of contracting:</p> JobServe</h3> https://www.jobserve.com/gb/en/Job-Search/</a></p> By far the most reliable source of leads. Slightly shonky website, bit fiddly to use but it seems that all the recruiters post their wares here.</p> Tactics:</p> Upload a pdf cv</li> Run a not particularly tight search (e.g. asp.net as a keyword with 50 mile radius, ordered by distance)</li> Blindly send your cv to all of them without worrying too much or bothering to address the post.</li> </ul> </p> This might sound lazy, but in my experience I almost never hear back about the advertised role, but what this does do is let recruiters that specialize in your particular skills know you are actively looking, which is a lot easier for them to make use of than hoping to tempt someone who isn't.</p> Workable</h3> https://jobs.workable.com/</a></p> LinkedIn</h3> By posting regularly the progress of my search and asking contacts for help I have had some luck getting referrals. I've never used LinkedIn's proper job posting system but it's probably worth checking out.</p> It's worth being careful if you are looking before leaving. I tend to not look till I've finished the previous contract and had some time to refresh so that hasn't been an issue for me.</p> LinkedIn job hunt status</h4> You can now set your "open to work" status to everyone, recruiters etc so be sure to set that. I've definitely had inbound interest from that, albeit relatively low quality. It also adds an overlay to your profile image which your network might see when you post things.</p> LinkedIn Hiring groups</h4> Outside IR35 contracts board</a> group. - I can do invites if you want them. Let me know.</li> </ul> Recruiters</h3> It's probably worth reaching out to recruiters directly.</p> They cold email us so no reason not to go the other way when you need a job. Could work out for everyone.</p> Here's a useful list that you could use as a source of leads as well as a way of filtering all the inbound email into a folder https://github.com/alexmbird/uk-it-recruiter-domains/blob/master/domains.txt</a></p> People you know (your "network")</h3> Possibly with the assistance of LinkedIn, WhatsApp, Telegram, Signal, Facebork etc.</p> I don't like to mention this as the idea of "networking" always made me cringe, but the truth is who you know matters and can get you an "in" where you might not have before, and indeed in the past I've won contracts through friends of friends and repeat business.</p> It doesn't seem very actionable, and when you need it it's probably too late to build a "network" of trust, but you can still reach out to the people you do know with a friendly message and make it clear you're looking (and be clear about what you're looking for - to trigger what Jonathan Stark calls "Rolodex Moments"</a>)</p> This is also a reminder to leave a trail of positive experiences with everyone you meet, go the extra mile, and invest a little regularly in your relationships whether you "need" them in the moment or not.</p> Others</h3> These are ones I've come across that you might want to cover if you are looking to not miss anything, but I haven't either used or succeeded with these myself. Suggestions for additions to this list welcome, just drop me an email or raise a PR</a>.</p> https://gun.io/find-work/</a></li> https://news.ycombinator.com/submitted?id=whoishiring</a></li> https://outsideir35roles.com/contract-listings/</a></li> https://remoteok.io/</a></li> https://snap.hr/</a></li> https://uk.indeed.com</a></li> https://wellpaid.io/</a></li> https://www.amazon.jobs/</a></li> https://www.contractoruk.com/it_contract_jobs</a></li> https://www.contractspy.co.uk/</a></li> https://www.cv-library.co.uk</a></li> https://www.cwjobs.co.uk</a></li> https://www.icontract.co.uk/</a></li> https://www.itcontractjobs.co.uk</a></li> https://www.itjobboard.net</a></li> https://www.jobserve.com</a></li> https://www.jobsite.co.uk</a></li> https://www.linkedin.com/jobs</a></li> https://www.monster.co.uk</a></li> https://www.reed.co.uk</a></li> https://www.technojobs.co.uk</a></li> https://www.upwork.com/</a> - seems to be low paid commodity labour on an international market largely, however I think some people have managed to break through this and become in demand. At that point you're basically an entrepreneur. https://www.quora.com/What-should-I-do-to-increase-my-earnings-on-Upwork-when-I-have-been-registered-on-it-for-over-a-year-and-have-done-only-one-10-job-for-which-I-received-a-five-star-rating</a></li> </ul> </li> https://www.yunojuno.com/</a></li> </ul> Deceased sites</h4> https://jobs.github.com/</a></del> - github jobs decommissioned</a></li> Sadly stackoverflow jobs has been decommissioned</a>. https://stackoverflow.com/jobs?j=contract</a></del></li> </ul> Places to find even more</h3> https://www.quora.com/What-are-the-best-job-boards-for-software-engineers</a></li> </ul> Practical tips</h2> Track everything you've applied to (Trello, or HubSpot if you're feeling rich).</li> Keep a copy of the full advert description and rate, they are often taken down before you speak to anyone which weakens your negotiation position.</li> Follow up with phone calls after applying - makes you more real, makes it easier for them.</li> Track all the businesses in your sector and technology, and their hiring managers and contact them before letting recruiters get in the middle.</li> See also my recruiters page</a> for defense against the dark arts. Or you could just blanket declare "no recruiters".</li> </ul> About this post</h2> Someone I know is looking for work (hello!) so rather than just emailing I thought I'd make a more permanent list of all the places I've come across for finding work as a contract developer.</p> For context, I'm a backend developer, specialising in Microsoft ASP.NET / C# & SQL Server databases, with a smattering of other things (t-shaped people as they say).</p> Setting up a static website/blog with jekyll 2019-06-24T00:00:00+00:00 A couple of people asked me for more info on my new blog setup here's a rough outline of what's involved in setup and posting. There are many</em> ways of doing this, this is just the way that suited me best for now.</p> What you need to follow along:</p> A domain from GoDaddy</a></li> An account on GitHub</a></li> </ul> Things I like about this setup</h1> Built-in backups,</li> offline editing,</li> no shonkey web gui,</li> no charge for hosting with custom domain and https,</li> simple dev friendly templating,</li> a nice simple code-friendly mobile-friendly design,</li> full control.</li> public revision history</a> of blog posts with no extra effort (no more "updated: blah" in blog posts)</li> </ul> Why jeykll fits that</h1> A blog post like this one is just a text file in a special format called markdown. These can then be trivially source-controlled with git, which serves as version control and distributed backup in one neat package. I use git for work so although it's not the easiest tool to use that's not a problem for me. Theoretically if you don't know git you could probably just do all your editing in the github web interface these days.</p> Here's the entire</em> folder structure of this blog as it stands right now. You can see there's really not a lot too it.</p> tim@fox:~/repo/timwise.co.uk(master)</span></span> $ tree</span></span> .</span></span> ├── 404.md</span></span> ├── about.md</span></span> ├── categories.md</span></span> ├── CNAME</span></span> ├── _config.yml</span></span> ├── _drafts</span></span> │ └── 2019-06-24-setting-up-a-jekyll-blog.md</span></span> ├── favicon.ico</span></span> ├── fonts</span></span> │ ├── aramisi.ttf</span></span> │ ├── EBGaramond-Regular.ttf</span></span> │ └── FuturaPTLight.otf</span></span> ├── images</span></span> │ ├── 404.jpg</span></span> │ ├── reverie-demo.png</span></span> │ ├── reverie.png</span></span> │ └── reverie-text.png</span></span> ├── _includes</span></span> │ ├── analytics.html</span></span> │ ├── disqus.html</span></span> │ ├── meta.html</span></span> │ └── svg-icons.html</span></span> ├── index.html</span></span> ├── _layouts</span></span> │ ├── default.html</span></span> │ ├── page.html</span></span> │ └── post.html</span></span> ├── LICENSE</span></span> ├── _posts</span></span> │ └── 2019-06-21-yet-another-new-blog.md</span></span> ├── _sass</span></span> │ ├── _darcula.scss</span></span> │ ├── _highlights.scss</span></span> │ ├── _reset.scss</span></span> │ ├── _svg-icons.scss</span></span> │ └── _variables.scss</span></span> └── style.scss</span></span> </span> 7 directories, 30 files</span></span></code></pre> The posts are just text files using the markdown format with a .md</code> file extension instead of .txt</code> and a short header for information like page title and tags. Here's the top of this one:</p> ---</span></span> layout: post</span></span> title: Setting up a static website/blog with jekyll</span></span> categories: [howto,backstage]</span></span> ---</span></span> </span> A couple of people asked me for more info on my new blog setup here's a rough</span></span> outline of what's involved in setup and posting. There are many ways of doing</span></span> this, this is just the way that suited me best for now.</span></span> </span></code></pre>How I set it up</h1> I actually am reconfiguring from another setup, and used some advanced sillyness, but I'll simplify that here down to roughly the steps you'd need to go from cold as I'm sure that's more interesting to you.</p> Buy a domain from GoDaddy</li> Create a GitHub account</li> Find a jekyll theme</a> on become a base for your new site (the themes vary on how ready they are for you to just crack on). Here's the reverie theme</a> that I used with minimal customisation.</li> Go into the settings for your repo, </li> turn on github pages, </li> configure your domain name, </li> enable https </li> Go to GoDaddy and set up the DNS to point to the github servers</a></li> </ol> Blogging with Jekyll</h1> Create new text file in the _posts</code> folder, name it yyyy-mm-dd-your-post-title.md</code>,</li> steal the header block from above and customise it for your post,</li> write stuff</li> use markdown for layout,</li> add images by saving them to an images folder and also adding them to your git repo, use ![alt text](/images/something.png)</code> to show the image in your .md</code> file,</li> check everything in & git push</code> to github,</li> your changes are live within seconds of pushing.</li> </ol> If you aren't up to the full git</code> setup, you can edit your blog directly in github's web interface.</p> Creating a new post </li> Editing a post </li> Editing the text of a post </li> </ol> Advanced ninja coder blogging</h1> Part of the reason I chose this particular setup is that plain markdown blogging with github pages allows me to use my existing coder tools to blog things really fast, reducing the barrier to me actually sharing useful insights and thoughts with you all.</p> You can see the advanced editing flow in action here:</p> https://www.youtube.com/watch?v=w3gMZTKcGKc</a></p> </iframe> Here's the tools I use, as shown in the above video:</p> zsh</a> - allows changing directory without cd</code></li> symlinks</a> to link blog folder into ~/blog</code></li> shell script new</code></a></li> template md file</a></li> jekyll</a></li> github pages</a></li> github actions</a> - automatically deploys pages (automatically set up when you enable github pages)</li> markdown</a> - this is important, it allows efficiently writing richly formatted blog posts with nothing but an (advanced) text editor such as vim or vscode</li> vim</a> the original text editor (see also neovim</a> that I haven't got around to using yet)</li> fzf the fuzzy finder</a> - so I can type *[tab]</code> and find any blog post easily</li> vscode</a> with the vscode vim emulator extension</a> - ctrl-p to open any post quickly</li> git</a> - provides history, and a very quick way to ship changes to the host (github) all on the command line</li> I have many command line aliases in my dotmatrix</a> and aliases in my gitconfig</a> - these make working on the commandline (cli) muuuch faster.</li> Ruby locally (advanced coder things)</li> asdf-vm</a> to ensure I always have the right ruby version available on any machine, or can easily install it</li> tmux</a> to allow me to have the server running in the background while I use the editor</li> </ul> ⚠️ Note: ignore the _site</code> folder when choosing the file to edit, that's the generated cache and will be overwritten.</p> Summary</h1> Wordpress was overcomplicated for my needs, and slower to use/read, this new setup should mean I can tap away on the train and send new thoughts your way more regularly.</p> Extras</h1> Bonus points for MailChimp's rss to email</a> which means it'll automatically hit your inbox. (I haven't manageed to make it work yet though.)</li> Here's all the jekyll gems github pages allows you to add https://github.com/github/pages-gem/blob/master/lib/github-pages/plugins.rb#L5</a></li> </ul> Blog refresh - bye bye wordpress, hello jekyll 2019-06-21T00:00:00+00:00 The old blog is gone. Move over wordpress, long live github pages</a>.</p> I haven't redirected the old subdomain or urls, but the stats tell me I had no visitors so meh. If you want something let me know and I'll repost it.</p> The new theme is : Reverie</a>.</p> The full history and source of this blog is available in the open at https://github.com/timabell/timwise.co.uk</a>.</p> Database tools I was surprised existed 2019-06-10T00:00:00+00:00 But you already know all the tools for working with databases don't you?</p> That's what I thought too before I started working on SQL Schema Explorer</a>. Even after 18 years working with databases it turns out I only knew a fraction of the tools that are out there.</p> Working on SQL Schema Explorer gave me a reason I didn't have before to search the internet in new ways relating to databases, and now that I'm reaching out to people about SQL Schema Explorer people are in turn sharing their favourite tools with me.</p> I'd hate for this gift to sit hidden away in my private product trello board, so I've turned it into this article so that you can skip the wilderness years and gain a zen-like awareness of all the tools of your trade. I'm sure you won't like or use all of them, and many overlap in function, but the following list will allow you to try out and asses each one against your needs; perhaps you'll find a new tool that will save you hours, or make your job that much more enjoyable. Never again will you have to say to a new colleague "oh I hadn't heard of that one!".</p> Microsoft Sql Server</h2> SSMS Tools Pack</h3> https://www.ssmstoolspack.com/</a></p> An add-on to management studio that improves many pieces of of ssms and adds new capabilities.</p> {:height="200px"}</p> €30 for first computer to €5000 for unlimited enterprise use (SSMS Toolpack Pricing page</a>)</li> </ul> A friend of mine saved many hours using the "CRUD" stored procedure generation capabilities.</p> Sql Server Management Studio (SSMS)</h3> Okay you know this one but I have to mention it.</p> It has awkward but functional diagram support. You can version control these diagrams and move them between servers with https://github.com/timabell/database-diagram-scm</a> which is worth knowing about if you ever use the ssms diagrams.</p> You can now run MSSQL on open source</h3> Did you know Microsoft SQL Server (aka mssql) is now available on both linux natively and in docker containers? It's the real deal, not like mono vs .net</p> SQL Server on Linux</a></li> SQL Server in Docker</a></li> </ul> Here's all it takes to fire up mssql, the only pre-requisite is docker itself.</p> docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=your_new_sa_passwod' \</span></span> -p 1433:1433 --name mssql1 \</span></span> -d mcr.microsoft.com/mssql/server:2017-latest</span></span></code></pre> This isn't a tool per-se, but this is too important to not mention, especially as it's quite new and not everyone is aware of this exciting change from redmond.</p> I don't know about you but one less reason to fire up the Windows VM sure does make me happy. Combined with dotnet core I haven't fired up Windows in months now.</p> Cross-database</h2> Razor Sql</h3> {:height="200px"}</p> http://www.razorsql.com/</a></p> "RazorSQL is an SQL query tool, database browser, SQL editor, and database administration tool for Windows, macOS, Mac OS X, Linux, and Solaris. RazorSQL has been tested on over 40 databases, can connect to databases via either JDBC or ODBC"</p> </blockquote> EZ Data Browser</h3> {:height="200px"}</p> http://www.softimum-solutions.com/Data-Browser/Overview.aspx</a></p> $29.99 per computer</li> Licence key by email</li> pay via PayPal of credit card</li> </ul> "software from Softimum Solutions to help users to browse and to edit SQL Server databases quickly and easily"</p> </blockquote> Provides a configurable MSAccess-like interface to a database.</p> SQLeo</h3> Cross-database query builder/analyser. Feature rich - data/schema compare, pivot queries, diagrams etc.</p> {:height="200px"}</p> http://sqleo.sourceforge.net/</a></li> </ul> "A powerful SQL tool to transform or reverse complex queries (generated by OBIEE, Microstrategy, Cognos, Hyperion, Pentaho ...) into diagrams to ease visualization and analysis. A graphical query builder that permits to create complex SQL queries easily. The GUI with multi-connections supports virtually all JDBC drivers, including ODBC bridge, Oracle, MySQL, PostgreSQL, Firebird, HSQLDB, H2, CsvJdbc, SQLite. And top of that, everything is open-source!"</p> </blockquote> SQLeo demo video on youtube</a></li> Sourceforge page (ugh) https://sourceforge.net/projects/sqleo/</a></li> Fork on github https://github.com/ojwanganto/SQLeo</a></li> </ul> SQL Fiddle</h3> Live online sql editor / runner</p> {:height="200px"}</p> http://sqlfiddle.com/</a></li> </ul> DbSchema</h3> {:height="200px"}</p> http://www.dbschema.com/</a></li> </ul> "DbSchema is an SQL and No-SQL database designer featuring interactive diagrams, HTML and PDF documentation, schema versioning and migration, relational data browse, random data generator, visual query builder, SQL editor and database reports."</p> </blockquote> DbPrompt</h3> {:height="200px"}</p> http://www.dbschema.com/dbprompt.html</a></li> </ul> "Free Universal Multi-Database SQL Prompt - DbPrompt can execute queries on multiple databases, transfer data between databases, upload result files on ftp servers, execute complex SQL scripts using Java Groovy and cron-schedule scripts for execution and report failures per email. DbPrompt supports all SQL and NoSQL databases, like MySql, Cassandra, PostgreSql, MongoDb, Redshift, SqlServer, Azure, Oracle, Teradata and more. DbPrompt can work on all operating systems. DbPrompt is free of charge. "</p> </blockquote> Unsorted - omg will it never end</h2> http://www.magereverse.com/</a></li> https://dataedo.com/</a></li> https://dbvis.com/</a></li> https://docs.microsoft.com/en-us/sql/azure-data-studio</a></li> https://github.com/ajdeziel/SQL-Data-Viewer</a> - abandoned</li> https://github.com/preston/railroady/</a></li> https://help.talend.com/reader/ISPDm8GQ6s0HN0348QulWg/Ij~7tBlW8im63rAGnGHT3A</a></li> https://portableapps.com/apps/development/database_browser_portable</a></li> https://redash.io/</a></li> https://sequelpro.com/</a></li> https://softwarerecs.stackexchange.com/questions/11346/tool-to-visualize-sql-database-schema</a></li> https://sqldbm.com/en/</a></li> https://sqlitestudio.pl/</a></li> https://www.codediesel.com/data/5-tools-to-visualize-database-schemas/</a></li> https://www.datasparc.com/</a></li> https://www.dbsoftlab.com/online-tutorials/active-table-editor-online-tutorials.html</a></li> https://www.devart.com/dbforge/sql/studio/</a></li> https://www.idera.com/er-studio-data-architect-saftware</a></li> https://www.jetbrains.com/datagrip/</a> - https://www.youtube.com/watch?v=Xb9K8IAdZNg</a></li> https://www.metabase.com/</a></li> https://www.navicat.com/en/products/navicat-data-modeler</a></li> https://www.schemacrawler.com/</a></li> https://www.sqlservercentral.com/articles/microsoft-sql-server-utilities-and-tools-1</a></li> https://jetbrains.com/</a> intellij/rider/rubymine etc all have the datagrip capabilities built in</li> https://www.red-gate.com/</a> sql prompt etc</li> https://www.oracle.com/database/technologies/appdev/datamodeler.html</a></li> https://erwin.com/products/erwin-data-modeler/</a></li> DBeaver</li> Azure Data Studio - PostgreSQL extension</li> SAP PowerDesigner https://www.sap.com/products/powerdesigner-data-modeling-tools.html</a> - I have no idea what this is. Sounds enterprisey. https://en.wikipedia.org/wiki/PowerDesigner</a></li> </ul> </li> https://www.idera.com/er-studio-enterprise-data-modeling-and-architecture-tools</a></li> https://www.idera.com/er-studio-data-architect-software</a></li> https://sparxsystems.com/products/ea/</a></li> https://www.ibm.com/us-en/marketplace/infosphere-data-architect</a></li> https://www.postgrescompare.com/</a></li> </ul> Places to find even more database tools and learn more</h2> https://en.wikipedia.org/wiki/Comparison_of_database_tools</a></li> https://www.quora.com/How-do-I-generate-an-entity-relationship-diagram-for-a-SQLite-database-file?share=1</a></li> https://www.quora.com/What-are-some-good-online-database-schema-design-tool-with-larger-days-of-expiry</a></li> https://alternativeto.net/software/mysql-workbench/</a></li> https://www.datasciencecentral.com/profiles/blogs/top-6-data-modeling-tools</a></li> https://www.educba.com/9-best-data-modeling-tools/</a></li> http://www.agiledata.org/essays/dataModeling101.html</a> - I had no idea what data modelling really meant before reading this.</li> http://www.sqlservercentral.com/articles/Tools/157911/</a></li> https://www.apgdiff.com/</a></li> http://toolsfordatabases.com/</a></li> https://www.webfx.com/blog/web-design/top-five-best-database-management-tools/</a></li> https://solutionsreview.com/data-management/data-management-solutions-directory/</a></li> https://www.softwaretestinghelp.com/tools/26-best-data-integration-tools/</a></li> </ul> Information overload</h2> If you got this far you probably need to get back to work. It's clear this article is a bit ridiculous now so I'm not going to carry on sorting it out. The internet is littered with catalog articles of varying completeness and quality. To make it complete and good would make it dull and endless.</p> Watch this space, I think I'll try and work out what people actually are trying to do and cater for that. Maybe we can all build a collaborative index somewhere.</p> The end</h2> I hope you found at least a few you didn't know about and that they make your life better in some way. Please do tell me the story of how this helped you on email</a> or twitter</a>. Did I miss something? If you wish to improve this article please ping me a PR with additions here: https://github.com/timabell/sdv-website or just email me</a>.</p> I'm not being paid to promote these, these are not affiliate links, I share this learning with you all for free so that we can all enjoy our work with databases more, and create better more reliable databases for ourselves, our clients and our projects.</p> If you want to be notified of new articles, sign up to the mailing list (which currently is also the trial download list).</p> Till next time!</p> Originally posted at http://schemaexplorer.io/blog/2019/06/10/database-tools-you-didnt-know-about.html</a></p> Teardown of a landing page story 2018-12-10T08:40:13+00:00 Here's my analysis of the depth of information and thought that's gone into a deceptively brief piece of landing-page copy. It's from https://realtimeboard.com/blog/accelerate-growth-early-stage-startups/</a> that I'm trying to use as inspiration for http://schemaexplorer.io/</a></p> I had to de-construct the example story in order to work out how to do my own so having put the work in I thought I'd share it with you all here.</p> Here's the original in full:</p> “Hey, I know that you have two kids and you’re always buying the same milk, eggs, peanut butter, and bread every single week, but you’re also both really busy working parents. You don’t have much time and you’d much rather be able to spend time at home with your family than being at a busy grocery store. So there’s this great service called Farmstead where you’re able to have that same stuff delivered to you. You don’t even have to think about it, you don’t have to remember each week to put in all of those same things that you order on a weekly basis anyway. They can just bring it to you and you don’t have to worry about it. Delivery is free with a weekly subscription. I think you should definitely try this out, I think you’re going to save yourself a bunch of time.”</p> </blockquote> And now one piece at a time with my observations about what is being conveyed in this information-dense block of prose. You are looking at the work of a pro when you read the above text, and unless you look closely it's easy to miss just how carefully crafted the whole thing is and how it touches on so many thoughts and feelings the ideal customer of Farmstead is likely to have, guiding them to the conclusion that they should give the service a try.</p> Hey,</p> </blockquote> being personable, we are not a big faceless corporation, you can actually talk to us like humans</p> I know that you have two kids</p> </blockquote> narrow down audience to parents (turn off non-parents, get parents more interested)</p> and you’re always buying the same milk, eggs, peanut butter, and bread</p> </blockquote> describe a behaviour they recognise they already do (increase engagement with the text, encourage reading further), hint that we're going to solve a problem they have because they might be annoyed about the regular chore</p> every single week,</p> </blockquote> slightly pejorative terminology hints at the fact you might not be happy about repeating this tedious task, draw attention to your current pain that we're going to solve</p> but you’re also both really busy working parents.</p> </blockquote> focus down the pain point to be that it's a time consuming repeating the task and you wish you had more time back, hint that we're going to be giving that time back to you</p> You don’t have much time</p> </blockquote> really drive that you have no time</em> point home</p> and you’d much rather be able to spend time at home with your family</p> </blockquote> paint a picture of how much better your life would be if you take on our solution to your pain. Point out just what you're sacrificing to go to the shops.</p> than being at a busy grocery store.</p> </blockquote> not only do you have the pain of lost time, but the experience of shopping is unpleasant (because it's so busy)</p> Now that you know it's you we want to help, we recognise your pain and we've shown we know how your life could be better...</p> So there’s this great service called Farmstead</p> </blockquote> Introduce the brand name. Tease. Read on to satisfy that completion-bias tendency in your brain that leaves you yearning to know what information you don't have about this "Farmstead".</p> Also the name itself conveys a lot of meaning</p> "Farm" - fresh produce, direct from the supplier "stead" - homely, small supplies, not industrial or full of chemicals</p> where you’re able to have that same stuff</p> </blockquote> allay the fear that you'll still have to go back to the old thing anyway because there's something we don't sell</p> delivered to you.</p> </blockquote> oh I see, so it's a door to door delivery service. Noted. The shape of the offering is starting to emerge.</p> You don’t even have to think about it,</p> </blockquote> We've worked really hard on all our UX (user experience), automated as much as possible etc. so this will be easier than what you do now and better than competitors can do</p> you don’t have to remember each week to put in all of those same things that you order on a weekly basis anyway.</p> </blockquote> now you know it's even got built in scheduled deliveries</p> They can just bring it to you</p> </blockquote> re-iterate delivery</p> and you don’t have to worry about it.</p> </blockquote> reminder that you do have have to worry about getting your normal shop done, speaks to pain of running out of the things because you forgot, worrying how you're going to fit it in your day. Differentiates new offering as being something that just happens meaning you never run out</p> Delivery is free</p> </blockquote> Objection handling - people don't want to pay for delivery when they could go to shops for nothing (even though it has a cost it's how people see it). People have been burned before by finding something good then being stung for expensive delivery. I would expect that this is a common objection/question that came back in earlier rounds of testing with users and it's been integrated into this prose to pre-empt the question before it's even been asked.</p> with a weekly subscription.</p> </blockquote> warm people up to the payments they'll have to make in a non-threatening way</p> I think you should definitely try this out,</p> </blockquote> nudge to take action. "Try" implies that it's not a commitment and you can change your mind, allaying fears of being locked in for a year</p> I think you’re going to save yourself a bunch of time.</p> </blockquote> point to hard cost-benefit for the service - you don't have time (as mentioned to start with, we know this person feels like they have no time because they are still reading after they identified with the first section). One last reminder of the key pain to the offering solves - lack of time</p> Let me know if that's helped you in any way.</p> If you happen to be someone that deals with databases and are keen to try new tools then please do me a favour and sign up to the http://schemaexplorer.io/</a> mailing list (you'll have to ignore my terrible marketing text till I'm as good as Brady).</p> ~ Tim</p> A Modern Digital Government Outsourcer 2018-10-30T14:19:26+00:00 I've done a few contracts for our dear government and an idea for a business is starting to form. Here is the first draft of these ideas as a starting point for discussion and improvement.</p> The problem as I see it</h2> From my somewhat limited perspective as someone who has worked on a few software projects for government, including having the privilege of a brief stint with GDS</a> (wikipedia page on GDS</a>), I can see a looming problem of capacity that I wonder if I can help with.</p> A change is happening to the way government runs digital projects (or entire services with a digital aspect I should say). This change was kicked off with the formation of GDS back in 2011, and has been rippling out</strong> across the many departments of government ever since.</p> The skills</strong> to run such projects are in ever increasing demand as more politicians and senior civil servants realise the effectiveness of the approach and are pushed into the approach by the controls made available to GDS through the Cabinet Office.</p> Some of the projects going on in government a resourced with existing internal people</strong> (many of whom are contractors) who have been moving around departments executing brilliantly in the true spirit of who GDS started, but there are only so many of these people to go around.</p> There are consultancies who "do agile"</strong>, but that is only part of the picture of spinning up a new digital-focussed project and having it succeed.</p> And then there's the big old outsourcers</strong> who have for decades been the go-to for large government IT. It strikes me that the old-world outsources struggle to meet the needs of the new world, as it's a fundamental shift away from outsource-all-the-knowledge / lowest-bidder-wins / change-requests-for-profit model that to be fair they have been incentivised to operate within.</p> All of this leaves ministers and civil servants who have seen how well it can be done wondering where on earth they can find the people they need to make a project a success. It's all very well having a service standard</a> that advocates a modern user-centred service but if you can't rustle up a team or outsourcer who can make that happen then you are stuck in the old world with broken, overpriced and inflexible IT systems that we are so used to.</p> Note that the GDS way is not a refusal to use outsourcers at all, rather it is a call to not be held to ransom by outsourcers by passing all skills and knowledge over to them wholesale and them being unable to control them.</p> A new breed of outsourcer</h2> Here I'm going to set out a rough design for a business that could help senior people in government deliver these services. Your feedback and ideas would be greatly appreciated as I cannot create this alone.</p> The business would take leadership</strong> in researching, designing, iterating and building services where a department currently lacks the capacity to do so but has the desire to build a modern service for some perceived user need. It would do this by supplying one or more people who truly understand the GDS-way, and will guide the service through the cycles of discovery and delivery.</p> The business would explicitly not</strong> try to capture ownership, insider knowledge or advantage over the department in order to gain control over the service. Instead the business would ensure that the department receives the benefit of learning that happens during the project, producing outputs that can be referred to in the long run, and ensuring that relevant people within the department are (with the assistance of the department) deeply involved in the process.</p> The business would attain speed of delivery by recruiting and training a team to be able to assist the department in conducting research and delivering services. This team would consist of mixture of senior people who can make sure the GDS way is followed as much in spirit as in letter, and a deeper pool of more mixed experience who can be guided by the senior team members in delivering effectively in a way that works for the department, GDS, and the users. The business shall build processes around this team that assist them in repeatably recruiting, training and scaling while retaining the core principles that work in the favour of government and our users.</p> The department will be encouraged (maybe even required?) to build internal capability (i.e. have enough smart skilled people) to retain the knowledge gained during the process of building a service (or perhaps learning that it doesn't make sense to build a particular service). The department will also be encouraged to actively recruit team members to bolster their ability to run the services in the long term and retain the knowledge gained as a service is created and evolved (perhaps for an appropriate fee to compensate for the investment in the people who pass through the business's doors).</p> The business needs to be a profitable enterprise in order to be a sustainable entity that can help our government in the long run. Suggestions for funding models most welcome!</p> The business will not retain any assets / intellectual property etc. would allow the government to be held to ransom by the business. The government will always have the option to take the whole service in-house, or choose another outsourcer. Support for transition will gladly be provided (for a reasonable fee). The government is encouraged to own its own services.</p> How are you different from public digital</a>?</h2> Public Digital https://public.digital/ are a business founded by the founding members of GDS, and are off transforming other governments and organisations from scratch. That's a lofty goal and I wish them well! My goals are more modest, simply to provide the capacity, processes and skills pool for government deliver on the vision and habit of delivery left in the wake of these powerhouses of change. I hope I can make even a thousandth the impact they have made.</p> How to get there from here</h2> Currently I'm contracting, I like the idea of slowly growing a team from here to there. I'm not in a position to just drop everything and gamble everything on success, but perhaps there can be a smoother transition. For now I just want to have as many conversations as possible, so get in touch</a> and let me know how you feel about the whole idea. Love it? Hate it? I want to know.</p> Get involved</h2> Drop me an email: tim@timwise.co.uk</a></p> Sign up to the mailing list: http://eepurl.com/dLU4Rs</a></p> Always add braces 2018-08-28T12:14:02+01:00 Small matter of code style that I keep coming across.</p> Whether to write</p> if</span> (</span>thing</span>)</span> Action</span>(</span>)</span>;</span></span></code></pre> or</p> if</span> (</span>thing</span>)</span></span> Action</span>(</span>)</span>;</span></span></code></pre> or</p> if</span> (</span>thing</span>)</span></span> {</span></span> Action</span>(</span>)</span>;</span></span> }</span></span></code></pre> We should always use the braces to avoid introducing bugs when modifying the code manually or through merge tools.</p> For the reasons behind this, read the following:</p> https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/</a></li> https://dzone.com/articles/omitting-braces-not-just-a-mat</a></li> Anatomy of a “goto fail” – Apple’s SSL bug explained, plus an unofficial patch for OS X!</a></li> </ul> 5 ways to make your database better - by Tim Abell 2018-07-10T00:00:00+00:00 [1] Documentation</h2> Shoot me okay, but maintenance of software is [insert large number here] times the cost of creation, especially with relational databases. You are a pro working for a client, you owe it to them to make it possible for them to have future staff (and yourself!) be as effective as possible. You put all that effort into figuring out why a column should exist and have that name, now share that knowledge before you move on to the next greenfield project</p> Redgate SqlDoc</a> is great for rapidly adding missing documentation.</li> SchemaSpy</a> generates static html sites making it easy to see what documentation there is (or isn't!) and share it with the team. It's free & open source (although a bit fiddly to set up and run). It has particularly nice clickable diagrams.</li> Dataedo generates static html sites & pdfs</a> as well, and is commercial and slicker than SchemaSpy</li> This gist for source-controlling ms_description attributes</a> gives you a two-way source-controllable / editable list of your documentation in SQL Server</li> SQL Schema Explorer</a> generates dynamic html sites making it easy to see what documentation there is and share it with the team.</li> </ul> [2] Refactor your database</h2> Migrations are a thing now. Use them. You refactor your code, why wouldn't you refactor your database? Stop leaving landmines for future people - misleading names, bad structures etc. Use the redgate tools (ready-roll etc), use your orm’s tools (EF migrations, active record migrations). Yes you have to deal with data, but it’s the exception not the rule that it’s going to take hours to run because of data volumes.</p> Redgate's SQL Change Automation</a> (formerly ReadyRoll</a>) is an opinionated tool for creating and running database migrations, it even generates Database Administrator (DBA) friendly pure-sql deployment packages. Very impressive!</li> Redgate's SQL Source Control supports migrations</a></li> I've been using EF Core migrations</a> recently and they work well. There are equivalents for all the major platforms.</li> </ul> [3] Enforce data integrity</h2> Does your app fall over if the data is bad? Databases have many powerful ways of enforcing the rules your code relies on: nullability, foreign keys, check constraints</a>, unique constraints. Stop the bad data before it even gets in there. Now your database is enforcing these rules your code doesn't have to handle violations of them when reading data because they'll never happen</p> [4] Integration testing</h2> You have an ORM. Great. You have unit tests. Great. But where the rubber hits the road and your code sends SQL to a real database it breaks at runtime more often than you’d like to admit because the generated sql didn't jive with the real database structure or data in some obscure fashion. Automate the creation/test/destruction of your db and run full end to end integration tests. I suggest automating from the layer below the UI to keep the tests fast. There are many techniques for keeping the tests quick but still realistic: do end to end smoke tests instead of individual pieces, use an in-memory database, use database snapshots</a> or the fancy sql-clone</a> tool from Redgate to make creation / rollback virtually instant. Can you pull realistic (anonymised) data from production? Better still, now you’ll catch a whole new class of bugs before they hit prod.</p> Here's a guide from Redgate detailing one way to do continuous integration testing with databases</a></li> </ul> [5] Make it visible</h2> Are the only people that can see the database structures the coders and DBAs? do the business owners, support people, Quality Assurance (QA) people find it a mystery? You should be just as proud of your database as you are of your code, by shining a light on this dark corner of your digital estate you can make it as good as it should be, not an embarrassing backwater. By sharing the database in an accessible form to the non-coders in your team you can help them be more effective in their jobs.</p> The html generated by SchemaSpy</a> can be shared on any webserver to let your whole team see your schema structures</li> SQL Schema Explorer</a> can be run on your network or cloud hosting (schema explorer is dockerized</a>!) to give your team easy access to both the schema and data within the database.</li> </ul> Combine these tools with a continuous integration system and you have easy access to the bleeding edge of your databases development.</p> Take action now!</h2> 1.Make a start on at least one of these improvements today. 2.Share this article with your team - get everyone motivated to improve. 3.Share this article on social media - help spread the word that our databases deserve better!</p> I hope this has inspired you to make an improvement in the often unloved underbelly of your applications.</p> What do you think needs improving in the way we deal with databases? What change did you make because of this? [Let me know!](mailto:articles@timwise.co.uk?subject=making better databases)</p> Originally posted at http://schemaexplorer.io/blog/2018/07/10/5-ways-to-make-your-database-better.html</a></p> Building on my own just for you and your database 2018-06-26T00:22:08+01:00 I'm not a big company, in fact it's just me. I haven't even paid anyone to do anything other than my accountants. I'm not against the idea of paying people, I just want to know I really need to first.</p> I see a gap in the tools available to database people; it's bothered me for years (decades actually, I built an in-house variation of this in Classic ASP in the year 2000, yeah I'm getting old, thanks for pointing that out). I'm not the only one who recognises this gap, someone I showed schema explorer to said "oh I knew someone who built something like that", this is great as it means I'm not delusional about there being a problem to solve. I find it hard to explain to people what this thing is, as there's nothing quite like it out there. I'm still not really sure who it's for (other than previous visions of myself). I want to put the time and effort in to make something that's so valuable that people are happy to pay for it, and the money that brings will in turn allow my work to continue on the tool, bringing benefit to more people.</p> The core of the idea was navigating data across foreign keys, after talking to people it's evolved into an all-round schema (and data) exploration tool (hence the addition of diagrams). My vision is to keep it a general-purpose database-centric tool, avoiding temptation to solve niche problems with it where it could take away from that well-formed-tool experience. I think I now have a piece of downloadable software that's basically useful as it stands; but that there's something (or many things) between where it's at now, and people starting to get value from it by using it with their own databases. I don't know yet what that is and I hope to find out through more conversations with database people. I certainly have plenty of ideas what it could be, but need to collect evidence rather than trying to fix non-problems on the off-chance. Especially with my even more limited time now that we've just had our second baby boy.</p> This is my current story, I hope you found it more interesting than the world cup ⚽.</p> http://schemaexplorer.io/</a></p> Building a Windows 10 Development VM from scratch 2018-04-01T20:10:54+01:00 Rebuilding windows dev VM from scratch, reminds me why people put up an old build for so long. What a painful process!</p> I did wonder if I could get away with mssql on linux, and tried the mssql docker image, which hosed my linux kernel</a>. That's too many yaks for one morning.</p> Windows Install</h2> Download a virtualbox image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/</a></p> Attach the vm disk in virtualbox</li> Fire up and run updates</li> Install Rider via jetbrains toolbox</li> Shut down the VM</li> Take a virtualbox snapshot</li> </ul> backing up a vm</h1> cd VirtualBox\ VMs/ </span></span> base=. </span></span> src=win10-2018 </span></span> ls -lh $src </span></span> du -sh $src </span></span> df -h . </span></span> tar -cpC $base $src -P \| pv -s $(du -sb $base/$src \| awk '{print $1}') \| lz4 >> $src.tar.lz4 </span></span> # 8.79GiB 0:00:43 [ 205MiB/s] [=====================================================================> ] 99% </span></span></code></pre> Gist for backing up with lz4</a></p> Updates! Updates! Updates!</h2> (as Ballmer once said?)</p> poke windows updates</strong> until it finally decides there's no more</li> reboot</li> poke windows updates until it finally decides there's no more</li> reboot</li> poke windows updates until it finally decides there's no more</li> reboot</li> Install virtualbox guest additions 5.1.34</strong></li> reboot</li> shutdown</li> backup again</li> </ul> </a> </div> The Snapshot Trials</h2> VirtualBox has a neat snapshot feature</a>, so you can try out a whole tree of attempts at installs then squash them into the final image or roll them back.</p> Take a snapshot as a starting point for trying out the eternally broken boxstarter script.</p> Boxstarter</h2> Boxstarter script, mostly working in win 10</a> now, but so much more to do on top that, including windows branch of dotfiles</a> (always needs work). Read the script to see what it configures and installs.</p> You'll want to watch this vid at at least 2x playback speed:</p> https://youtu.be/Hiz9_i67B3o</a></p> annoyances / setup - grab reg keys</h2> disable screensaver</li> hide task manager when minimized HKCU\Software\Microsoft\Windows\CurrentVersion\TaskManager\Preferences</code> - binary. sigh.settings > m</li> </ul> </li> multitasking > snap (these have been rolled in to my boxstarter, but not tested there). HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced\SnapAssist</code> - 0 HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced\JointResize</code> - 0</li> </ul> </li> all sounds off - generates 100s of reg keys, one for each event</li> mute</li> </ul> get ready for dev</h2> vs sign-in</li> extension updates ssdt - external download popup, installs sql server 2016 localdb, takes forever, requires reboot</li> azure - leave for now, also external</li> </ul> </li> jetbrains toolbox (sign in) resharper</li> datagrip</li> </ul> </li> </ul> VS Extensions</h2> cinst editorconfig.vs vsvim ihateregions -y</code> - bit-rotten, didn't work - I should contact the VsVim package maintainer</a>. Oh never mind that would be me then.</p> Extensions to install manually:</p> https://marketplace.visualstudio.com/items?itemName=JaredParMSFT.VsVim</a></li> https://marketplace.visualstudio.com/items?itemName=ZoltanKlinger.RelativeLineNumbers</a></li> https://marketplace.visualstudio.com/items?itemName=MadsKristensen.MarkdownEditor</a></li> https://marketplace.visualstudio.com/items?itemName=VisualStudioProductTeam.VisualStudio2015ColorThemeEditor</a> - to get solarized colours</li> </ul> Something to look in to https://chocolatey.org/packages/batch-install-vsix</a></p> Git setup</h2> GitExtensions defaults to the program files putty, but the chocolatey one is newer and github deprecated the comms the old one uses.</p> https://github.com/desktop/desktop/issues/4105</a></li> https://blog.github.com/2018-02-23-weak-cryptographic-standards-removed/</a></li> </ul> Change all the putty paths in git extensions to start with C:\ProgramData\chocolatey\bin\...</code></p> ![Workspace 1_193]({{ site.baseurl }}/assets/workspace-1_193.png)</p> https://git-extensions-documentation.readthedocs.io/en/latest/settings.html</a></p> Settings that are specific to Git Extensions and apply globally will be stored in a file called GitExtensions.settings</code> either in the user’s application data path or with the program. The location is dependant on the IsPortable setting in the GitExtensions.exe.config</code> file that is with the program.</p> </blockquote> i.e .C:\Users\tim\AppData\Roaming\GitExtensions\GitExtensions</code></p> Key Setup</h3> Map a read-write folder to outside the VM to keep the putty keys in (this avoids regenerating the key if the VM is regenerated / rolled back, and also means keys aren't copied around with the VM image which makes them a bit easier to keep track of</li> git extensions > tools > putty > generate</li> set passphrase</li> save pub & private keys to shared folder</li> grab the public key from the generator, paste into github & bitbucket website account ssh configs</li> git extensions > tools > putty > start agent > add keys</li> </ul> DotMatrix</h3> Start git bash from git extensions, not the start menu (to get ssh set up right).</p> cd /c/repo && mkdir tim && cd tim </span></span> git clone git@github.com:timabell/dotmatrix.git </span></span> cd dotmatrix/ </span></span> git checkout windows </span></span> bin/install </span></span> git st </span></span></code></pre>VS Settings</h2> ![Workspace 1_194]({{ site.baseurl }}/assets/workspace-1_194.png)</p> VS > Options > Environment > Import/export > set the path to the dotmatrix copy</p> resharper settings</h2> ![Workspace 1_195]({{ site.baseurl }}/assets/workspace-1_195.png)</p> VS > Resharper > manage options > right-click the gap > add layer > open settings file > select the one from the dotmatrix</p> More settings</h2> VsVim settings</li> git config</li> email/name</li> </ul> More installs</h2> redgate tools</p> dev bundle: sql prompt - for VS too</li> compare</li> data compare</li> doc</li> </ul> </li> toolbelt bundle readyroll</li> </ul> </li> </ul> firefox</h2> sign-in</li> ghostery settings - disable annoying purple dot</li> </ul> My Screencasting Setup 2018-03-17T18:56:12+00:00 I just made a video to demo schema explorer.</p> </iframe> I did this in one take with no post-production work. I figure practice will improve my selection and delivery of the content.</p> Behind the scenes I used the following:</p> Hardware</h2> Dell XPS 13 - (9350 model), using the built-in webcam (up-nose view, camera in bottom left of lid for some reason), good enough for tiny pic in corner.</li> Snowball usb mic</a> (great sound quality, no worrying about analogue stuff), sat on a folded towel on my desk to deaden vibrations through the mini-tripod. The snowball has 3 settings on a switch at the back, had it on 3 - omni directional, I tested all three modes</a> beforehand.</li> Supplied stand didn't actually fit properly, so bought an ammoon mini tripod</a> instead which is lovely (importantly it has the right screw thread for the mic).</li> A quiet room after everyone is in bed to avoid interruptions</a>, and did the recording between rain storms!</li> </ul> Software</h2> Linux Mint</a> 18.3 Cinnamon</li> Vokoscreen</a> 2.4.0 - this handles screen recording & the picture-in-picture from the webcam</li> </ul> I recorded the entire screen at 1920x1080 (seeing as even phones are HD now). and uploaded it to youtube. (drag-n-drop, simplez!)</p> Settings</h2> Control panel > sound > input > snowball > about 130% input volume</p> ![Sound_183.png]({{ site.baseurl }}/assets/sound_183.png)</p> fullscreen capture, display 2, 1920x1080</p> ![Screenshot from 2018-03-17 18-27-39.png]({{ site.baseurl }}/assets/screenshot-from-2018-03-17-18-27-39.png)</p> pulse / blue-snowball selected</p> ![Screenshot from 2018-03-17 18-27-55.png]({{ site.baseurl }}/assets/screenshot-from-2018-03-17-18-27-55.png)</p> mpeg4, mp3, avi, 25fps</p> ![vokoscreen 2.4.0_180]({{ site.baseurl }}/assets/vokoscreen-2-4-0_180.png)</p> Default config apart from save path "videopath"</p> ![vokoscreen 2.4.0_181.png]({{ site.baseurl }}/assets/vokoscreen-2-4-0_181.png)</p> Webcam switched on</p> ![vokoscreen 2.4.0_182.png]({{ site.baseurl }}/assets/vokoscreen-2-4-0_182.png)</p> Other attempts</h2> I spent ages finding the right software and doing test takes to get sound levels etc right.</p> I started off with Kazaam for recording and mplayer for the webcam onscreen video.</p> I got mplayer to show a borderless webcam video picture on screen that I could then use with any screen recorder, but had problems with dual screen behaviour so even though I could zoom in better I gave up on that once I'd discovered vokoscreen.</p> You might also want to look at https://wistia.com/soapbox</a> which I hear is very good.</p> Regression Tests for ASP.NET / SQL Projects 2018-03-13T15:25:46+00:00 I've had the opportunity to attempt many variations of regression testing on many projects.</p> Many of the systems I've worked on have at least some level of workflow, meaning a user has to move through many steps as the system progresses something towards a final state.</p> As systems become more and more complex it becomes harder to not accidentally break an area of the system you weren't currently looking at and didn't re-test manually before shipping. Hence the search for the holy grail of the perfect regression test suite. It turns out that none of the approaches available (unit tests, UI drivers etc.) provide a single solution to the problem, and instead we have to use all these approaches in balance to give reasonable protection against regressions.</p> Here I lay out my current thinking on the best way to use testing to avoid regressions in a standard ASP.NET MVC + SQL Server project. I would like to introduce to you the idea of "business logic tests" which are an integration test suite starting below the UI layer, continuing through to a real database. I believe this gives an ideal balance of the trade-offs we have to make when choosing how to design out full regression test suite for a system.</p> System layers</h2> The layers of such a system are ideally something like this:</p> Database (largely just used for CRUD storage)</li> ORM - EF or similar library</li> Model - C# classes</li> Business logic layer - pure C# - (often missing in real projects, with business logic scattered across the other layers)</li> UI layer - Controller classes written in C#, views in razor .cshtml files, html, css, javascript etc.</li> </ul> Recommended test layers</h2> Unit tests</strong> - test small pieces of C# code</li> Business logic tests</strong> - test the business logic layer down through to a real SQL Server database.</li> UI Smoke tests</strong> - non-exhaustive test of a few common user journey's through, running against a real database, driven with Selenium Web Driver or similar.</li> </ul> There should be only a few smoke tests, there should be many business logic tests covering all business functions that need to stay working, and finally developer discretion on the amount of unit testing to apply at the lowest level.</p> What do the business logic tests look like?</h2> These could be pure code driven through something like NUnit, or they could be in business language through a BDD tool like specflow.</p> They depend on the business logic project but nothing above that (UI). They inject a real database connection as a dependency through whatever injection method you use.</p> These tests should make sense to the product owner / client.</p> This is the layer with the broadest coverage of all of the three styles in play. Every business use-case, workflow & variation should be covered here. This is the suite that will alert you if any business or workflow rule is no longer being applied as designed, or if an obscure step of an obscure branch of your workflow has stopped working (perhaps due to an odd interaction with a database constraint).</p> Handling the database</h2> SQL Sever has a very useful snapshot feature that means you can very rapidly roll back to a clean state after each test run. (Snapshot setup script</a>)</p> For each business logic test run:</p> Reset to the production snapshot</a> (or production equivalent build if you can't actually grab a production backup)</li> Run outstanding migrations for this version of the code (you are using Octopus + ReadyRoll</a> aren't you?)</li> Run the Business logic tests.</li> </ol> Why not test all the business functions through the UI?</h2> UI tests are prone to reliability problems, you can end up with 98% pass rates being normal, so then you don't notice when a critical test fails.</li> UI tests are slow - this can add an extra overhead to developer productivity.</li> UI tests cause unwarranted resistance to change - once you have UI tests the UI becomes extremely hard to change, particularly for sweeping site-wide redesigns without spending a prohibitively long time fixing up the test suite. And you aren't fixing the test suite because you changed the logic, this is busywork.</li> The UI isn't sufficiently fragile to warrant this level of testing - when you are working on a page you are unlikely to break unrelated pages without realising.</li> </ul> I have seen first-hand a team of excellent programmers get bogged down in problems with a full regression test suite driven entirely from the UI.</p> Why not test all layers separately?</h2> In my experience a major point of fragility when a system is modified is the interaction between the code and the database. As such it's important that this fault-line is covered thoroughly by all regression tests that focus on business functions that need to remain functional.</p> A test suite that isolates the business logic layer from the database layer provides very little value, usually just fulfilling a "though shalt unit test all classes</em>" declaration.</p> How to get there</h2> If you don't have a solid business logic layer then you will need to refactor your way to that goal. This has other benefits for reliability beyond the ability to test. Clean code with an obvious layer for all the business logic is easier to modify and keep defect-free.</p> Bonus - free test data</h2> The tests that you write to drive the business logic layer will be generating real data in a real copy of your database as it runs through the many steps in a workflow. You can use this test driver to generate useful test data for every step in a workflow, e.g. if a workflow had four steps you could do the following:</p> Test account A - drive business logic to step 1</li> Test account B - drive business logic to step 2</li> Test account C - drive business logic to step 3</li> Test account D - drive business logic to step 4</li> </ol> This is extremely quick to run, and now you have an account in every state of your workflow. This is useful for easily:</p> Demoing your system from any starting point with multiple test accounts.</li> Having the data in the right state to do development work on state 4 without having to manually set it up.</li> Give your manual testers all the test accounts they need in many interesting states.</li> Testing out your reporting systems and anything else that consumes data from your database.</li> </ul> Having realised you can do this, why would you ever do it manually again?</p> Because the code that generates this data is a core part of your regression test suite it will be maintained as part of normal development activities and so you'll never again have to deal with test data that no longer works with your latest version of the system.</p> If you want me to help you get to this ideal then drop me a line - tim@timwise.co.uk</a></p> Developers, Love Your SQL Database 2018-03-12T19:30:15+00:00 Developers, don't be afraid of your SQL database, don't try and ignore it, try and make it the best it can be just like you do for your code.</p> What's wrong today</h2> I've noticed that a lot of projects that have SQL databases don't apply the high standards that they have for their codebase (refactoring, clean code, documentation, etc.) to the SQL part of their projects.</p> Dealing with relational databases is harder than code:</p> The tooling that I see teams using to manage their relational databases is crap.</li> Having to handle live data makes refactoring harder (but not impossible).</li> SQL is a language from the stone-age of software.</li> </ul> But that's no excuse for not doing the best you can for the problem at hand.</p> There are answers to these problems</h2> Some of the answer is just accepting that it's harder and learning to be effective. For example I find VsVim</a> and SQL Prompt</a> to be great for editing SQL.</p> Redgate</a> have a great suite of tools for managing the full lifecycle of a database. Use them. I particularly like the way ReadyRoll</a> handles migrations. RedGate tooling has excellent integrations with Octopus deploy</a>.</p> So if you're in the situation I've seen with a database you virtually can't change and nothing but a visual studio database project in place that may or may not be in place with production, then try the following:</p> Throw away the database project.</li> Set up ReadyRoll in your visual studio project.</li> Grab your production schema, and use that as a base in your ReadyRoll migrations. (It has a neat feature for a first "migration" that will only be run on a new database build, so it can build you a dev database from scratch, but it won't attempt to recreate production).</li> Configure ready-roll to use branch folders.</li> Start creating migrations in feature branches.</li> Set up CI / test / QA / pre-prod environments (or whatever you call them in your team) to drop and rebuild the database with every build. You can make this faster by using SQL Server snapshots.</li> Each environment then runs the database migrations that match the version of the codebase that was released to that environment.</li> Push changes up through the environments until those migration scripts have been run so many times you are no longer afraid to run them.</li> </ol> Now you can refactor your database with confidence just like you can with the code, and it can stop being something you pretend isn't there and start being something you are proud to show off.</p> There are many potential complications, such as re-indexing tables with lots of data in production, but these are not things that an intelligent well-functioning DevOps team can't handle well once the basic process is in place.</p> Further Reading</h2> https://gist.github.com/timabell/6fbd85431925b5724d2f</a> - source control your schema documentation (ms_description attributes)</li> https://dba.stackexchange.com/questions/515/how-do-you-document-your-databases</a></li> </ul> If you like this, you might appreciate the tool I'm working on to shed further light on the dark corners of your SQL database, check it out at http://schemaexplorer.io/</a> and be sure to sign up to the waiting list.</p> My approach to my work 2018-03-10T00:00:00+00:00 I've realised that something that sets me apart from other developers with similar buzzwords on their CV is the way I think about things and the principles I think are important. But I've never put them on my CV or tried to express them in writing before. I'm not perfect and may fall short sometimes but these are the perfection I strive for.</p> Tell the Truth, Even if it's Hard</h2> I don't believe in selling good news only and robbing people I work with of the negative information that could be critical to making good decisions. E.g is this project going as well as expected?</p> No. - opportunity to take corrective action by those with the authority to make sweeping changes</li> Yes (a lie) - temporary happiness followed by reality catching up and being in an even worse situation</li> </ul> Great episode on the matter: The Startup Chat with Steli and Hiten: 288: Startup Lies https://thestartupchat.com/ep288/</a></p> Strong Opinions, Weakly Held</h2> Which means I have strong opinions about what the right thing to do is based on my experience and the information available at the time, but that if I am presented with new information then I am open minded to changing my opinion. I first heard about this while on contract with GDS, an organisation with an amazing culture for doing the right thing and getting things done. Here's a small example: I spent many many years preferring the C# style of skipping curly-braces for one-line if</code> blocks, based on the understanding that the less boiler-plate there is the easier it is to read the code. And then a fellow dev argued that it was better to always include them for the sake of reliability. I was initially unconvinced, but I always appreciate hearing a contrarian view and I proceeded to research extensively the pros and cons. Given the constant problems with indentation (spaces/tabs) in many projects that can make an if block look like something it isn't if you aren't paying close attention, and the automatic merge present in many source control systems, I was persuaded to change my view and will now argue just as strongly for the new stance</a>. For an excellent example of a serious security problem that might have been avoided by always having braces, read the "explanation of Apple's goto fail SSL bug</a>", a coding error which has the hallmarks of an unfortunate choice made by a merge algorithm.</p> Communicate to be understood</h2> I've had feedback that my communication with stakeholders is good, and that I can explain complex technical issues clear to less technical people. I think it's important that those around a project such as senior stakeholders, users, related support people and collaborating suppliers all have the best understanding of the state of a project and any issues. Without clear information in a language they are comfortable with then they are left unable to help the project succeed. I think it's important not to dumb things down as that results in losing important details, but instead take the time to help people understand the details they need to and why they might be important. As the technical experts it's up to us to make sure that others have the knowledge they need to do their jobs the best they can.</p> Help others</h2> I've also had feedback that one of my strengths is mentoring and sharing knowledge with other team members, technical and non-technical. It's certainly true that I do love helping others improve their own skills and knowledge where I can, and sharing what I've learnt over the years (ahem, decades). This is partly just because I love what we can do with technology and want everyone to gain the maximum benefit from it.</p> Continuous Personal Improvement</h2> I read the 7 habits of highly effective people</a> by Covey many years ago, and it is truly life changing because it changes your thinking from fatalistic "this is how I am" to "I can become a better person", and gives you some great mental tools for doing so. I have practised this consciously ever since. I particularly liked the quote I heard recently "What would you tell your younger self? Nothing, they wouldn't be able to understand what I know now, I'm a different person</em>" (or words to that effect).</p> </a> </div> Follow Programming Best Practice</h2> Our industry has learnt a lot over the last 50-odd years about what's effective. A lot of best practice has been captured in books, internet articles, and importantly libraries and frameworks. Strive to stand on the shoulders of the giants.</p> Elegance, Practicality, and Definitely No Dogmatism</h2> It's easy to get drawn in to the latest hip thing and apply it to everything you do no matter how inappropriate. Every situation should be assessed for the trade-offs that will be appropriate. One of the challenges of defining best practice in programming is the variety of problems it is used to solve. Should we use the same rules to produce a life-support system with no ability to push updates out as we should for a startup SaaS with no users yet? Clearly not, yet you'd wonder given some of the views presented; usually of the form "This worked on my project so you MUST now use it in ALL YOUR CODE!!!</em>" (This is the way Test Driven Development was introduced to me, and eventually I realised this is probably because it came largely from the dynamic language Ruby that lacks all the compile time checks I'm used to in C#, and as such complex systems have more failure modes as the codebase changes over time, resulting in a greater value of extensive unit testing.)</p> Emergent Architecture</h2> Planning is good, don't get me wrong. But don't restrict your options until you need to. The idea of emergent architecture is do the simplest thing that will work (following the You Aren't Going to Need It principle, aka YAGNI), and then as it becomes apparent you have a need for a particular abstraction or pattern that's when you apply it, through the magic of refactoring. I've seen too many projects that have been made many times more expensive than necessary because at an early point in the project a developer who had previously been burned by a lack of some structure or other made damn sure it was in their next project, regardless of whether it was actually necessary for the problem at hand. Another classic is learning of a pattern and applying it without it actually being needed. A common example of this I've seen is adding a "repository" layer to a codebase on top of entity framework, not realizing that entity framework already implements the repository pattern</a> (along with the unit-of-work pattern). I spent years learning about the various approaches to the architecture of software projects. Initially I thought I need to apply all of this best-practice to every project, but was troubled by how long that would take and found myself guiltily doing just enough to get the job done.</p> Technical Debt and Refactoring</h2> Refactor problematic things as they cause you problems. Simple. I really can't say it better than this: https://ronjeffries.com/xprog/articles/refactoring-not-on-the-backlog/</a></p> The Agile Thing</h2> I learned about eXtreme Programming (XP), scrum & kanban (I've actually worked in a real factory that used kanban). I've worked in many teams at various levels of competence on the whole "agile" thing and to me it seems to be down to good people making the best use of the available ideas for the problems at hand. Got no process at all, or something horribly like a waterfall? Add some elements of scrum and get better. Nailed scrum and feeling constrained? Do some more reading up, decide where to go next. I don't pretend to be an expert in "agile", but I love working in effective teams making use of the best methods available, and I love helping teams improve where I can. I'm not particularly interested in climbing the management-track ladder, I'm happy to work as a member of great teams. You can't do it all, and I don't want to get too far from my coding skills.</p> Resources</h2> These are things I've read that have guided my thinking towards where it is today (along with my personal experiences).</p> Books</h3> Mythical Man Month</a> - a timeless classic</li> </ul> Refactoring and technical debt</h3> https://ronjeffries.com/xprog/articles/refactoring-not-on-the-backlog/</a></li> http://laughingmeme.org/2016/01/10/towards-an-understanding-of-technical-debt/</a></li> </ul> Compassion</h3> https://jml.io/pages/your-code-sucks-and-i-hate-you.html</a></li> </ul> Technical details</h3> http://endoflineblog.com/gitflow-considered-harmful/</a></li> </ul> New thinking</h3> Things that I've recently come across that I wish to share / promote</p> https://simpleprogrammer.com/agile-is-dead-code-review/</a></li> </ul> Get In Touch</h2> If you need someone who you can rely on to bring more than just a list of keywords to your project then get in touch: tim@timwise.co.uk</a></p> Contract developer to entrepreneur - resources 2018-02-16T22:31:38+00:00 I'm not there yet so I'm no authority, but here's some things that I think are fab resources on my journey. They are helping me change my thought patterns and behaviour to make success more likely.</p> Books (roughly in order of my journey)</h2> The e-myth (revisited)</a></li> A better mousetrap - the business of invention</a></li> The Art of the Start</a></li> Key person of influence</a></li> So good they can't ignore you</a></li> </ul> More books over on my goodreads profile</a>.</p> Some more suggestions on the cocoon blog</a></p> Inspiration</h2> https://www.softwarebyrob.com/2015/03/26/the-stairstep-approach-to-bootstrapping/</a></li> </ul> Podcasts</h2> Bootstrapped Web [Website</a>] [Feed</a>]</li> Marketing for Developers [Website</a>] [Feed</a>]</li> Noah Kagan Presents [Website</a>] [Feed</a>]</li> Rogue Startups Podcast [Website</a>] [Feed</a>]</li> Startup Chat [Website</a>] [Feed</a>]</li> Startups For the Rest of Us [Website</a>] [Feed</a>]</li> The Art of Product [Website</a>] [Feed</a>]</li> The Indie Hackers Podcast: How Developers are Bootstrapping, Marketing, and Growing Their Online Businesses [Website</a>] [Feed</a>]</li> The Startup Chat with Steli and Hiten [Website</a>] [Feed</a>]</li> The Tim Ferriss Show [Website</a>] [Feed</a>]</li> Tropical MBA - Entrepreneurship, Travel, and Lifestyle [Website</a>] [Feed</a>]</li> Zen Founder: Startup. Family. Life. [Website</a>] [Feed</a>]</li> </ol> My feed list</a> exported from AntennaPod</p> Products</h2> github-pages</a> - for hosting landing pages etc</li> logojoy.com</a> - $65 automated and very good logo generator</li> godaddy - for domains</li> </ul> Education</h2> http://tinymarketingwins.com/</a> - learn marketing even before you have anything to sell</li> https://designacademy.io/</a> - design for developers</li> https://www.ditchinghourly.com/</a></li> http://startupclass.samaltman.com/</a> - also on youtube & as a podcast:</li> </ul> https://www.youtube.com/watch?v=CBYhVcO4WgI&list=PL5q_lef6zVkaTY_cT1k7qFNF2TidHCe-1</a></p> Community</h2> Indie Hackers</a></li> </ul> People</h2> Noah Kagan</a></li> Tim Ferris</a></li> Courtland (Indie Hackers)</li> Laura Elizabeth</a></li> Jonathan Stark (Ditching Hourly)</li> </ul> Down with SDV! long live SSE! 2017-12-29T02:59:54+00:00 I can't keep up with myself on this startup journey, but I hope I can give you a peek into my journey as I go. I've had couple of revelations recently which I'll explain here:</p> Discovering your needs</h2> I've managed to talk to more people about what problems I could solve and have started to learn more about what value this tool might provide (solution-to-problem instead of the usual solution-looking-for-problem).</p> I had originally thought of this tool as something for viewing data, but I've had several people express an interest in the ability to visualize relationships. This was surprising to me as personally I've been using SqlHawk</a> (my fork of SchemaSpy</a>) to get to grips with database relationships, but I hadn't considered that they don't get widespread use in the circles I move in; presumably due to a mix of awareness and what a pain they are to set up (Java, shudder).</p> Having explained my vision for the tool to my friend and ex-colleague Ben, I managed to ask him to say in his own words how he might hypothetically describe it to others, and he uttered the magical words: "well it's kind of a schema explorer". And suddenly I understood that the real value of the tool I'm creating is not in the ability to poke around the data per-se, it's actually the ability to come to an unfamiliar relational database and be able to much more quickly understand how it hangs together. (It'll have value in familiar databases too, but I like this focus, it feels higher-value.)</p> As a result "diagrams" is now the top-priority feature being added to SDV.</p> It also gives me a new name for Sql Data Viewer...</p> </a> </div> New name, new domain</h2> The product has been renamed to Sql Schema Explorer</strong>, which can now be found on the shiny new domain schemaexplorer.io</a></p> For the new website:</p> I've given up on all the fancy wordpress designs and hosting</a> I was using for the old domain and have gone for github pages</a>.</li> All the bloated css and javascript is gone, leaving nothing but a hand-crafted flat html</a> file that should load beautifully on any connection thanks to being under 1kb of content in total with no trackers or external resources.</li> It should reliably render on most devices with minimal CPU load or delay thanks to the lack of bloated and fragile javascript / device detection etc.</li> I've ditched the fancy drip sign-up form in favour of a simple email link (why solve the volume problem when I don't have it yet?).</li> I've ditched the google analytics tracker because it's a vanity metric if it's not money in the bank; and these trackers have hidden costs (page load time, leaking your customer's data to google etc). If I get a sudden influx I can just ask them how they found me rather than reading between the digital lines.</li> I've stripped the content back to almost the bare minimum to try and drive focus towards email signups so that I can then engage in-depth through that. I need enough that I can point people at it in conversation to get them signed up. I only need a dozen early customers to help me shape the product so I don't need to optimize for complete strangers who've never heard of me yet.</li> I've created my own personal style of design that favours minimalistic css and markup to keep page load fast and complexity down whilst still giving structure to the message and a little bit of easiness on the eye (e.g. off-white/black, read the css if you like). I'll be continuing to improve my design skills</a> as I work on this project, though I'm not against outsourcing when I need to.</li> </ul> Money, money, money</h2> You may remember I wrote about making SDV free</a> not too long ago. With the above new understanding about the value that the product can provide and what it will take to build a full-featured product for that market I am no longer concerned about being immediately copied, and think it has sufficient value to be a viable product in its own right. As such I will be charging for it after all.</p> I'll be following the great advice from the episode of Startup Chat - 268: Encore Episode – How to Get Your First 10 Customers</a> that I heard recently (highly recommend this if you're also thinking of getting a startup / side-project off the ground).</p> The starting price point will be £150 per single user/machine. This is based on the "3x the first number you think of" formula because I think they are right that first instinct is off; after-all I have to make this a viable business otherwise development will stop and no-one wins.</p> I'll be offering the recommended money-back guarantee for my first customers; that way I can prove that there is real value in what I'm creating whilst maintaining my high ethical standards for fairness in business.</p> Still here?</h2> Great! Thanks so much for reading this. If you're interested in keeping up to date or being involved in the early life of this new product then head over to schemaexplorer.io</a> and sign up!</p> From contractor to entrepreneur - motivations 2017-11-19T23:01:08+00:00 I'm not sure what changed but I used to be happy publishing open source code on github and thinking I'd done the world a mighty favour. But slowly I started to become uneasy. How much good have I really done if I publish something that only a couple of people find, let alone benefit from? I also started to find it harder to find any time to spend on software to give away for free in between the day job and my growing family. I think that might be why I now find myself pursuing for-profit side projects. It's not everything but revenue is some measure of the value I am creating for others. Revenue is also a way of spending more time creating that value; every pound I earn from my creations is a pound I don't have to earn from the day job, and because I'm a contractor I have the luxury of being able to adjust my work levels. If I achieve any success with my own projects that would allow me to invest more of my time in them and generate yet more value without sacrificing family life.</p> Ever since I entered the world of work I've wanted to create things that help people in some way. I think one of the reasons I still love software development is the joy of watching someone realise I've made one small piece of their life or job better, easier or faster in some meaningful way. Creating my own products that gain wide adoption is the ultimate expression of that for me.</p> I've already had some fantastic unexpected benefits of starting on this journey. It's reminded me to be humble, and given me even greater respect for anyone who has created a business. It's pushed me to read and listen to many things I wouldn't have thought of before, and reading with a goal in mind seems to be a much deeper learning experience. It's enabled me to have fascinating conversations with people I otherwise wouldn't have had a reason to talk to, such as the lovely generous people at grow@greenpark.</p> I've discovered huge holes in my skills that I was oblivious to in my cosy enterprise-developer role, such as marketing, and I am now super-motivated to learn more.</p> Maybe I can build something that you would find useful? What's the biggest hole in your digital life?</p> Please sign up for my mailing list</a> so we can follow this journey together.</p> Looking for beta-testers for my new Free Sql Data Viewer tool 2017-11-18T01:28:08+00:00 Hi, my name's Tim and I'm a UK based developer trying to build my first startup. I'm working on a data-browsing tool for Sql Server etc. that I really believe can help people do their jobs.</p> I'm reaching out to the online communities to try and find people who will find it useful and who are prepared to help me shape the early product.</p> Please take a look at my offering on https://sqldataviewer.com/</a> ... and if you like the look of it then please sign up to my waiting list</a>.</p> Thanks for your precious time!</p> Tim Abell</a></p> Why SDV could be a billion dollar business 2017-11-17T23:10:48+00:00 Okay maybe a billion is a long shot, but I read the Hiten Shah article on how Trello could have been a $1B business</a> (via startup chat</a>) and was surprised to discover there are lessons for how I can turn Sql Data Viewer</a> into a sustainable business.</p> What trello and SDV have in common:</p> Mass (horizontal) appeal</li> Easily copied</li> Difficult to sell premium versions of the general purpose product ("stickers" anyone?)</li> </ul> I was going to just sell SDV for £100 and be done with it, but this article showed me a better way.</p> Give the general purpose tool away for free (as Trello does)</li> Collect contact information in exchange for the value provided by the free tool.</li> Use the contact list to have conversations with businesses that use it in order to...</li> Find out how integrating tightly with their business could provide real value over and above the general purpose tool.</li> Build vertical-specific integrations, stickyness & marketing.</li> Profit! (No underpants required. Or gnomes.)</li> </ol> What do you think? Drop me a line: tim@timwise.co.uk</a></p> Git for TFS users 2017-10-27T20:44:06+01:00 I'm considering creating a 30 minute video to help teams transition from Team Foundation Server source control to git (hosted with TFS or Visual Studio Team Services - aka VSTS).</p> Please register your interest</a>. If I get enough interest in this then I'll do the work to put it together and launch it.</p> ![tfs-to-git-terminal]({{ site.baseurl }}/assets/tfs-to-git-terminal.png)</p> Choosing wordpress hosting for a new idea 2017-10-22T11:32:08+01:00 I need somewhere to keep the marketing site/content for my new Sql Data Viewer</a> product. (Since renamed to SQL Schema Explorer</a>.</p> I could use github pages</a> or something similar, but I've found the overhead of needing a dev environment to make any change puts me off getting things done. I've been using wordpress.com for this blog and am pretty happy, especially having a mobile app. I've also heard using wordpress makes it easier to outsource content/design if you have enough success to make that worthwhile.</p> I don't fancy maintenance, security patching and backup, so the marketing site is currently on wordpress.com</a></strong> where I've paid for the "personal" plan at £3/month</strong> to remove ads and the £0.92/month</strong> for mapping a domain. Affordable for contactor running a shot-in-the-dark startup side-project that may never make a penny. Note that you only get their built in analytics, you can't add google analytics, and you can't install plugins on that plan.</p> To get google analytics you need "premium" at £7/month</li> To be able to add plugins you need "business" at £20/month</li> </ul> This is on top of the £1.15/month</strong> for domain registration and £10/month</strong> for a VPS (virtual private server) to run my demo site from, both from the excellent bytemark</a> hosting company, who've been fab on support from day one.</p> Just for completeness: the demo site connects to sql azure instances are easily within the $40/month included with my MSDN Professional subscription that I use for contracting. Handy.</p> I've been listening to lots of startup podcasts (email me</a> if you want the list), and have discovered the mighty Justin Jackson</a> and his excellent Marketing for Developers</a> content - I recommend you buy it if you're a developer thinking of selling a product / service. I'm now working through trying to apply the lessons in the book to my own product. I got to the analytics bit and hit a wall. Wordpress.com (the wordpress hosting site, not the software itself) force you to upgrade to the business plan at £20.83/month</strong>. Given I don't know how long this is going to take me I'm not keen to start ramping up costs to that extent with no obvious route to a return on the investment.</p> I've looked at the three recommended in Justin's book (plus more added more recently), and the starting prices are:</p> Siteground</a>: Startup plan (like it!) £2.75/month</strong> - looks like this might be a winner then! Oh wait, that's a sign-up discount, it's actually £6.95/month</strong>. Hrumph.</li> Krystal</a>: Personal £18/month</strong></li> WPEngine</a>: Personal $29/month - about £22/month</strong>.</li> Pagely</a>: VPS-1 $499/month - about £380/month</strong> - ouch! I don't think I'm their target market. I wonder if they repriced, perhaps Justin should drop this one from the book.</li> </ul> And that's where I'm at right now. Maybe I'll just shove a docker wordpress image on the VPS, it's already a docker host. I need to figure out hosting multiple sites on one IP for the demos anyway. But then again, I don't want to be a wordpress sys-admin. Or maybe I'll rethink the static site thing.</p> Home server docker-compose 2017-08-29T21:26:28+01:00 Another step in making a throw-away home server: docker-compose.</p> Find my compose file and setup script at https://github.com/timabell/home-server-docker-compose</a></p> I learned how to use docker-compose while working on reverse-proxying sdv, find out more: https://github.com/timabell/sdv-docker</a></p> Home server backups 2017-08-29T20:31:08+01:00 The setup</p> Ubuntu server domU xen host Ubuntu server xen VM with LUKS full disk encryption docker-compose syncthing with built-in "Staggered File Versioning"</li> </ul> </li> </ul> </li> </ul> </li> </ul> Plan</p> Plug in usb external hdd, with full disk encryption (i.e. a LUKS partition taking up almost all the space, plus a little fat32 with a text file in case anyone finds it and wants to return it).</li> Use LUKS key chaining to be able to unlock the disk without entering a password</li> Use autofs to automatically mount/unmount so that it's safe to unplug when a backup isn't running.</li> Use rsync to push all the files from the syncthing data directory onto the usb disk.</li> </ul> Somehow need to make the disk accessible to the VM.</p> Finding the luks partition when plugged into the host:</p> tim@spot:~$ sudo fdisk -l</span></span> ...</span></span> Disk /dev/sdc: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors</span></span> Units: sectors of 1 512 = 512 bytes</span></span> Sector size (logical/physical): 512 bytes / 512 bytes</span></span> I/O size (minimum/optimal): 512 bytes / 512 bytes</span></span> Disklabel type: dos</span></span> Disk identifier: 0x6380ad37</span></span> </span> Device Boot Start End Sectors Size Id Type</span></span> /dev/sdc1 63 21579 21517 10.5M c W95 FAT32 (LBA)</span></span> /dev/sdc2 21580 1953147527 1953125948 931.3G 83 Linux</span></span> </span> tim@spot:~$ ll /dev/disk/by-uuid/</span></span> ...</span></span> lrwxrwxrwx 1 root root 10 Aug 29 21:51 6ca09b72-8c9b-4571-8943-9f1d520671ab -> ../../sdc2</span></span></code></pre> Confirm it's the luks partition:</p> tim@spot:~$ sudo cryptsetup luksDump /dev/disk/by-uuid/6ca09b72-8c9b-4571-8943-9f1d520671ab</span></span> LUKS header information for /dev/disk/by-uuid/6ca09b72-8c9b-4571-8943-9f1d520671abVersion: 1</span></span> Cipher name: aes</span></span> Cipher mode: xts-plain64</span></span> Hash spec: sha1</span></span> ...</span></span></code></pre> I'll update this post with any details as I progress. Don't hold your breath though!</p> Running sdv in docker 2017-08-13T20:35:14+01:00 Just bought a shared server with bytemark</a>. (£10/month), Installed vanilla ubuntu 16.04LTS server using the control panel (virtually one-click).</p> ![Servers - Bytemark Manager - Mozilla Firefox_143]({{ site.baseurl }}/assets/servers-bytemark-manager-mozilla-firefox_143.png)</p> (Love the cancel button text!)</p> Ssh'd in, created my own user to use with sudo instead of root. Ran the following, and immediately had a copy of sdv listening on the internet.</p> sudo apt install tmux docker docker-compose</span></span> sudo adduser tim docker</span></span> # logout & reconnect to get new group to take effect</span></span> wget https://raw.githubusercontent.com/timabell/sdv-docker/master/docker-compose.yml</span></span></code></pre> and then for the magic:</p> tim@sdvweb:~$ docker-compose up</span></span> Pulling sdv (timabell/sdv:latest)...</span></span> latest: Pulling from timabell/sdv</span></span> d5c6f90da05d: Pull complete</span></span> 1300883d87d5: Pull complete</span></span> c220aa3cfc1b: Pull complete</span></span> 2e9398f099dc: Pull complete</span></span> dc27a084064f: Pull complete</span></span> eb1a4736b68c: Pull complete</span></span> 0706cf350247: Pull complete</span></span> 1d0ac78e96a5: Pull complete</span></span> Digest: sha256:9003a79f019b3ee16e7c6324afd275b4535f867602e63db317e430528e2a6771</span></span> Status: Downloaded newer image for timabell/sdv:latest</span></span> Creating tim_sdv_1</span></span> Attaching to tim_sdv_1</span></span> sdv_1 \| ./sdv-linux-x64 -listenOn 0.0.0.0 -port 8080 -driver sqlite -db /data/Chinook_Sqlite_AutoIncrementPKs.sqlite</span></span> sdv_1 \| 2017/08/13 19:14:44 Sql Data Viewer v0.4; Copyright 2015-2017 Tim Abell <sdv@timwise.co.uk></span></span> sdv_1 \| 2017/08/13 19:14:44 ## This pre-release software will expire on: 2017-10-01 00:00:00 +0000 UTC, contact sdv@timwise.co.uk for a license. ##</span></span> sdv_1 \| 2017/08/13 19:14:44 Starting server on http://0.0.0.0:8080/ - Press Ctrl-C to kill server.</span></span></code></pre> And that's it, listening on the internet!</p> ![Sql Data Viewer - Mozilla Firefox_144]({{ site.baseurl }}/assets/sql-data-viewer-mozilla-firefox_144.png)</p> Any changes I make will be to the docker and compose files</a>, resulting in trivially easy to repeat deployments, making the server completely throwaway. Huzzah! Docker is awesome.</p> I've also bought a domain for the product, but haven't set much up on it yet: http://www.sqldataviewer.com/</a></p> Next up, getting nginx reverse proxy to provide ssl. And finishing the refactor I was in the middle of to get data types to behave, and finishing the automated regression tests I'd just started (needed for reliable multi rdbms support), and finding my market, and marketing the product, and doing more features, and build the marketing automation to drive sales, etc etc. Not too much to do then! Your support would be appreciated, if you're interested make sure you sign up to the mailing list now</a>!</p> If you're wondering why docker matters more generally then listen to this podcast episode: Hanselminutes: Practical Containers for Developers with Aja Hammerly http://www.hanselminutes.com/default.aspx?ShowID=18514</a></p> Patreon open source funding 2017-06-03T10:24:40+01:00 I'm now on patreon</a>.</p> Hopefully this will provide enough funding that I can take better care of the open source projects that I look after. Since becoming a daddy it's become hard to find the time to do free work on these projects even though I love working on them. Combine that with the ambition to create a startup which eats up the rest of my spare time and the result is that these projects that I love are getting stale.</p> As a contractor my time is available for purchase. Unlike in permanent employment I can vary the amount of paid client work I take on in a year, so if I get serious funding for the open source work I can legitimately scale back my normal client work to make time for it. I'm not looking for people to buy me a pint out of gratitude here (though I wouldn't complain!), I'm looking for serious funding from people and organisations that benefit from all the hard work put in so that we can all benefit from further improvements to the great public codebase that is open source.</p> As an example, my ef-enum-lookup nuget package</a> has had 18k downloads to date, and I suspect this is unlikely to be used in hobby projects given the nature of EF (entity framework, an object-relational-mapper from Microsoft). So a fair number of commercial operations are benefiting from the improvements to their database that this project brings. I've had several requests and even some pull requests for improvements and features that I would dearly like to add, and that these businesses would benefit from; if they pooled money via Patreon then it would actually be an economical way for them all to benefit from the improvements. For example I counted six different individuals expressing an interest in custom schema support</a>. What if they or their companies all contributed?</p> If you want a project to not dissolve into a mess then even pull requests are not free. Although there is a probably-working improvement, it may not fit the style of the project, it may introduce other issues, it may be missing unit tests, it may be written by a less experienced programmer and need additional work and/or support.</p> I've had to choose monthly or per-feature on Patreon, and I've gone for monthly because there's more to an open source project than just bug-fixes and features - you also have to package it, unit test it, host it, refactor it, document it, set up continuous deployment or whatever the flavour of the day is, and respond to people who are having issues or who have contributions to offer. It seems to be a better match to the nature of open source development to ask for ongoing support as once someone is using a project they are likely going to benefit from using it for some time, and even a new feature will have a long life providing value to users.</p> I haven't picked a specific daily rate I'd work on this for yet, but I'd probably give some discount over my normal contract rates just because I think it's worthwhile and I enjoy it.</p> Do you or your employer use any of my open source projects? Fund me to make them better! Head over to https://www.patreon.com/timabell</a> with your boss's credit card and then drop me a line to let me know what's of interest to you.</p> Dustbin of ideas - IT Contractor Buddy 2017-05-12T12:32:07+01:00 I've talked to a couple of people experienced in the recruitment industry and it appears that the contractor buddy</a> business idea has some fatal flaws and is unlikely to succeed.</p> You might be surprised to hear that I consider this a success on my journey to startup success.</p> I've had a business idea (not an easy thing to do!) which is more realistic than many of my previous ideas showing that my ability to spot such opportunities is improving. (It has a clear revenue model and its fairly easy to see how it could be built.)</li> I've not become too attached to it to hear valid criticism.</li> By putting it out there and discussing it openly with anyone willing to help me out I've rapidly found the flaws in it without wasting valuable time or money. (Imaging if I spent two years building it to then have the same flaws sink it.)</li> I've managed to find contacts from a broad range of backgrounds who have helped me to crystallize my idea - this is entirely thanks to spending time at the fabulous co-working and startup community space grow@greenpark</a>.</li> </ul> So what sank it?</p> Even though the company would not be inserted into the financial transaction, they would still need to be a communication relay between employer/agent and candidate, and the received opinion is that this just won't fly.</li> The number of contractors who could find this useful could be small, with the highly in-demand people not having to put in much work to get the next contract (therefore not gaining much value from the service), and the the not-in-demand contractors are not a market the business would want to be primarily associated with as it could become a by-word for low-quality talent. This leaves the middle ground which may not be enough of a market to support the business. Before I'd talked to anyone I'd assumed that all contractors would want this (based on sample size of one - me).</li> </ul> </a> </div> The investment required to prove/disprove these suppositions would be significant, as I think you'd basically have to build the functioning business to see what happens in reality. So on that basis, "I'm out!"</p> So what now?</p> Well there's my Sql Data Viewer</a> that I haven't given up on (check it out, let me know what you think and join the mailing list).</p> There's also the possibility of a pivot</a>. Maybe there's room for something to help contractors without getting in the middle? Maybe it could send mail using their real email address and handle responses? Maybe there's already something like that out there and I should just use it instead of trying to invent my own! Let me know in the comments if you know of anything or already have ways to ease the pain.</p> Thanks for listening.</p> IT Contractor buddy 2017-05-03T13:31:51+01:00 A new job-hunting assistant for contractors</h2> Calling all IT contractors; would you pay to have some of the pain of contract hunting taken away?</p> I'm considering launching a business for IT contractors like myself that will do the tedious and time-consuming bits of job hunting.</p> If this is something you think you'd be interested in then sign up to the mailing list for early access</a>.</p> Concept</h2> Search job postings for you, letting you know when good matches come up, across multiple job boards.</li> Submit your CV on your behalf</li> Keep track your applications (when you sent your CV where for what rate).</li> Handle first contact from companies / recruiters, validating it's worth your time</li> A simple monthly fee, with no long-term commitment.</li> No cut of your contract rate and no fee per placement.</li> </ul> A concierge service for IT contractors.</p> As I know from first-hand experience the job hunt is a time-consuming slog where being on top of all your leads is critical to avoid being taken advantage of rate-wise by a sharp-eyed contingency recruiter, or having a deal ruined by duplicate CV submissions.</p> </a> </div> How is this different from all the other recruiters?</h2> In house recruiters are agents of the hiring company, and you may have to deal with many of these. We are working for you, the contractor, no matter how many companies and agencies you are talking to. They can only help you with respect to their company.</p> Contingency recruiters (i.e. 3rd party recruiters) are trying to insert themselves into the deal between you and a client, sometimes with the okay of that client, sometimes without. Unlike them we are not trying to skim money off your contracts, instead we are providing a service that helps you avoid the painful bits of the job hunt, for a straight-forward monthly fee that reflects the value we provide to you. Value for value. These recruiters will not help you with leads from other recruiters, in fact letting them know about other leads is a recipe for disaster.</p> So neither of these will help you with your overall job hunt. You still have to manage all your leads yourself.</p> How is this different from a talent agent?</h2> You may have read the new-yorker piece about</a> the agent 10x</a>.</p> They charge 15% of your rate</a> (a pretty big cut imho!) whereas we charge a flat rate for the service we provide; value for value.</li> They are limiting themselves to the mythical 10x'er. So even though they offer a lot of the concierge I'm offering, you probably can't get them to take you on. We however will take on even non-rocket-scientist contractors as we scale, free from BS hype about "rockstars".</li> Our service is clearly working for you and no-one else whereas with 10x's percentage fee it's not clear who they work for no matter what the marketing says.</li> 10x are attempting to find clients directly, just like every other contingency recruiter. We don't mind where your work comes from as we aren't trying to insert ourselves into the deal, so you won't be limiting your market by working with us, in fact we'll make it easier for you to cover more of the market by doing the hard work for you.</li> </ul> Sign up now!</h2> If you think this is something you might use, then please sign up to the mailing list</a> so you can get first access. There will initially be limited spaces for clients so that we don't overstretch, so sign up now</a> for first-come first-served early access.</p> New home server with Xen and docker 2017-03-10T00:03:48+00:00 No cloud mkay?</h2> Going against the current received wisdom I personally am not keen to trade the convenience of the public cloud sync services such as dropbox, onedrive, google drive etc for the fact that this means that all my files traverse the public internet and live on someone else's metal. Sure it might be encrypted, but what if my traffic is intercepted and stored, and then that encryption is later found to be flawed?</p> I'm probably just as likely to get pwned running my own setup, but hey at least I get to learn something along the way.</p> Another thought is that regardless of location - home, office, cloud, vps host or data centre - it's all just (server) software. So why shouldn't we be able to get the amazing seamless experience when self hosting? It seems to me that it comes down to economics. Cloud hosting gives an opportunity for charging directly or worse using underhand tactics like owning your data or selling you as a product to advertisers. This means there is more funding for cloud systems, and therefore more developers are working on it. Even ethically minded software developers have to eat and live somewhere. I think just like Linux vs Windows we will get a perfectly good open source solution, but it will likely lag behind.</p> Disk encryption</h2> I have a new machine to use as a server so have an opportunity for a fresh approach. I want my server encrypted at rest in case the whole machine gets stolen from the house to keep the scoundrels from having easy access to all my files.</p> The previous machine I installed with encrypted LVM as available with Ubuntu server's installer; but this means going to the machine to unlock it after every reboot / power failure / kernel upgrade. I did have remote unlock via dropbear ssh</a> set up but it's hacky to set up and non-trivial to use.</p> This time I've gone for a different approach:</p> An unencrypted install of ubuntu server (DomU) with an ssh server, so no password required to boot up, which does nothing more than give access to:</li> A Xen VM install of Ubuntu server (Dom1), this one with LVM disk encryption. This is where all my files etc will live. Installed with virt-manager</a>, which is a very convenient UI for initial setup.</li> </ul> So now after a reboot I can ssh -X</code> into the DomU server, then using virt-manager I can see the Dom1 VM waiting for the disk encryption key and provide the passphrase.</p> Xen</h2> Setup instructions for xen on Ubuntu</a>.</p> It wasn't immediately obvious how to set up the networking. This was the network config that worked in the end (after ifdown/ifup, also tested with full reboot):</p> $ cat /etc/network/interfaces</span></span> source /etc/network/interfaces.d/</span></span> </span> # The loopback network interface</span></span> auto lo</span></span> iface lo inet loopback</span></span> </span> auto xenbr0</span></span> iface xenbr0 inet dhcp</span></span> bridge_ports enp1s0</span></span> </span> # The primary network interface</span></span> auto enp1s0</span></span> iface enp1s0 inet manual</span></span> #iface enp1s0 inet dhcp</span></span></code></pre>Docker</h2> Seeing as docker is well and truly coming to windows it's time I got better at this so, for this and also better isolation and security I'll try and set up as many of the services I want to run as I can through docker.</p> First test case was transmission-bt</a> which I run to be a good citizen and help seed Linux Mint ISOs</a> using my A&A unmetered upload</a>.</p> Searching for transmission on docker hub</a>, I didn't want the vpn one, I'm not pirating movies and my ISP doesn't do stupid filtering/shaping tricks on torrents. I chose the next most popular: https://hub.docker.com/r/linuxserver/transmission/</a></p> To install and run it:</p> $ sudo -i</span></span> # useradd -r -s /sbin/nologin transmission</span></span> # id transmission</span></span> id=999(transmission) gid=999(transmission) groups=999(transmission)</span></span> # cd /var</span></span> # mkdir transmission</span></span> # cd transmission</span></span> # mkdir config downloads watch</span></span> # chown transmission:transmission .</span></span> # docker create --name=transmission \</span></span> -v /var/transmission/config/:/config \</span></span> -v /var/transmission/downloads/:/downloads \</span></span> -v /var/transmission/watch/:/watch \</span></span> -e PGID=999 -e PUID=999 -e TZ="Europe/London" \</span></span> -p 9091:9091 -p 51413:51413 -p 51413:51413/udp \</span></span> linuxserver/transmission</span></span> # docker start transmission</span></span> # docker logs -f transmission</span></span></code></pre> useradd ref: http://askubuntu.com/questions/29359/how-to-add-user-without-home</a></p> Once that was done I could see the transmission web UI at http://dom1vm:9091/</code>. Hurrah.</p> syncthing</h2> https://store.docker.com/community/images/linuxserver/syncthing</a></p> Same deal with folders and users, then:</p> # docker create --name=syncthing -v /var/syncthing/config/:/config -v /var/syncthing/data/:/data -e PGID=998 -e PUID=998 -p 8384:8384 -p 22000:22000 -p 21027:21027/udp linuxserver/syncthing</code></p> Start on boot</h2> docker update --restart=unless-stopped transmission</span></span> docker update --restart=unless-stopped syncthing</span></span></code></pre> http://stackoverflow.com/a/37479753/10245</a></p> Todo</h2> nextcloud</a> - has an iOS app that syncthing doesn't yet, might also have better selective sync https://store.docker.com/community/images/wonderfall/nextcloud</a></li> </ul> </li> plex server https://www.plex.tv/</a></li> </ul> </li> </ul> Further reading</h2> https://forum.level1techs.com/t/dexter-kanes-ultra-paranoid-encrypted-nas-completed/98340</a></li> </ul> New year new blog 2017-01-17T23:36:00+00:00 So-long blogger and thanks for all the fish.</p> My content to date will live on here for as long as google keep the bits spinning.</p> Find my new blog at http://blog.timwise.co.uk/2017/01/17/new-year-new-blog-happy-2017/</a> which will live for as long as I pay wordpress.com's bills and don't screw up my DNS config.</p> See y'all on the other side. Don't forget to subscribe to the new RSS feed.</p> Startup competitors - Data Viewer 2017-01-17T01:55:27+00:00 If you follow my ramblings you might know that I've revived my interest in creating a commercial tool to help devs, dbas etc browse their sql databases. I've done some research before and didn't find much on a par with what I have in mind. I've done a bit more, and turned up a couple:</p> A feature of sql workbench</a> - I've seen the product but didn't know it could do that. Not directly competition as it's not nearly as smooth to use as what I'm building for this particular use case.</li> SqlSmash</a> - Something I'd not seen before, which has fk based navigation (vid)</a> that's basically what I'm building, albeit embedded in SSMS rather than standalone.</li> </ul> This isn't the first time I've had the experience of getting started before discovering another product that basically solves the problem I'm gunning for. (Ready-roll</a> was the last, since bought by red-gate).</p> Now it's not that I thought my idea was magic or special or a unique little flower, but why is it after working with SQL Server databases for 15 years that I've only just noticed this tool (which I would have found useful and probably would have paid for).</p> If I created another would it be just as obscure? Should I still go ahead even though it's a solved problem? Are there any ideas left on the earth?</p> Something for me to consider this week. I might update this post with further thoughts. Your thoughts are welcome.</p> You might also like this post: Database tools I was surprised existed</a></li> I did end up building it, and it became https://schemaexplorer.io/</a></li> </ul> New year new blog, happy 2017 2017-01-17T01:00:09+00:00 I've used blogger for years, but the layout for code has not improved for a long time. I've also found my domain's website stagnating because of the hassle of updating it, so I've moved the whole kaboodle to wordpress.com</a>.</p> Find my old blog and all its content at http://timwise.blogspot.co.uk/</a></p> I'm not migrating the content from blogger because I think google will keep it online, and cool URIs don't change</a>. I've never been convinced of the value of people lugging their blog posts from one platform to the next, it seems to break more than it fixes most of the time. Like the one where the old host didn't need the trailing slash at the end of the URL and the new host did, breaking half the incoming links. It's only because I'm a web dev that I guessed what the fix was.</p> Enough about that anyway. Happy new year :-)</p> Please do add my new blog to your favourite blog reading software / rss reader etc., and do drop me a line on tim@timwise.co.uk</a> just so I know you're out there!</p> The original of the header is a photo I took myself which you can find on flickr: https://www.flickr.com/photos/tim_abell/17111067690/</a></p> javascript dates in firefox and locales 2016-09-27T07:31:00.002+00:00 Today I learnt:</p> You actually have to reinstall firefox using a different download to get new Date(dateString)</code> to use a different date format (i.e. non-US).</p> https://jqueryvalidation.org/date-method/</a></p> This method should not be used, since it relies on the new Date</code> constructor, which behaves very differently across browsers and locales. Use dateISO</code> instead or one of the locale specific methods (in localizations/ and additional-methods.js).</p> </blockquote> So if you're using chocolatey or boxstarter like I am (my boxstarter script</a>) you need to add the locale flag: https://chocolatey.org/packages/firefox</a></p> choco install Firefox -packageParameters "l=en-GB"</code>.</p> Reinstalling the package with --force</code> is sufficient to change it.</p> It would seem chrome has no way change the format from en-US</code>.</p> To be clear the correct solution is to do as the documentation says and not use that method, however it's a bit mean having a pitfall like that in the API. And sometimes you just have to work with the code you have...</p> Yet another good-commit-messages post 2016-03-18T15:25:00+00:00 Why you should care about writing good commit messages</h2> It is extremely likely you won't be the last person to touch this code.</li> You will forget the details of exactly why you did something after a few weeks/months/years. Be kind to your future self.</li> Source control systems out-last [access to] ticket trackers by many years (Jira, Github-projects, Trello boards, etc. etc., how many have you seen in your time working on code? A lot more than you've seen source control systems I'll wager.) A link to a ticket is not enough. Copy the relevant context from the ticket into the commit message.</li> Not everything can be deduced from the code. Even with excellent variable/function/class/module names, and beautiful refactoring, the circumstances that let to a particular design or change are lost. Good comments adding context help, but sometimes you don't want to clutter code with temporally relevant comments; the source control gives you a suitable place to keep this information locked up with your patch of the day.</li> When someone wants to know if they can delete a line of code in the future, if they can't fathom why it was put there then the only way to find out is to delete it and see what breaks. By adding context in a commit message they can use the source-control history to get that context, and then decide whether that context is still applicable.</li> You might think your code is perfect and correct and needs no explanation beyond the code itself; but what if there's a bug? Now the only documentation is the buggy code. If I come to fix your code later how do I know what it was supposed to do without going back to first principles. What algorithm or design pattern were you trying to implement? What references did you use?</li> Chesterton's Fence</a> - understand why something is the way it is before you change it.</li> And finally, I have seen first hand</em> developers being judged in their absence</strong>, on the quality of their git logs. For example when deciding whether to let someone go who is considered sub-standard for the team, the principal engineer and an engineering manager might go over the individual's output in private, without that individual present, to collect some evidence and inform their final decision. Do you want that to be you? Honestly, that "wip" commit with your name & email on it that seemed so harmless at the time doesn't look so good now it's out of context of whatever rush / justification there seemed to be in the moment. Worse still if you make a regular habit of lazy / rushed / incoherent commits.</li> </ol> "Write every commit message like the next person who reads it is an axe-wielding maniac who knows where you live" ~ unknown</p> </blockquote> Why the business owner cares about good commit messages</h2> (Even if they don't know it)</p> Developers writing high-quality commit and pull request (PR) messages provides clear context and explains the "why" behind changes, directly impacting the clarity and long-term value of the codebase as an asset.</p> Well-crafted messages ensure that future developers, including new hires, can quickly understand the rationale behind decisions, saving time, reducing the cost of onboarding and reducing the risk of misinterpretations that could lead to costly errors.</p> They support smoother collaboration across teams, foster accountability, and create a historical record that aids in audits, troubleshooting, and strategic decision-making.</p> This practice minimizes the risk of long-term inefficiencies, speeds up development, and ensures technical work aligns with broader business objectives.</p> Why I care about good commit messages</h2> For reasons of chance I've ended up more than once maintaining and extending things created by other people, many of whom had moved on from the projects and were no longer contactable.</p> When dealing with a piece of code that has a behaviour that is clearly causing problems for users/customers etc. sometimes I have needed to understand why</em> it was that way before I could know whether it could be changed without creating even bigger problems.</p> Another example was a buggy and complex algorithm implementation, but no clue left as to what the algorithm being implemented was, as a result a week of reverse-engineering the maths happened that could have been saved with a simple "this is an implementation of ..." with a hyperlink or algorithm name.</p> With the author of the code not available to ask I'm left only with the source code and git blame</code> to fathom why it's like that.</p> When you find the commit that added the patch and it's one of the following then you realise the author was not taking into account future maintainers:</p> "wip"</li> an entire fully formed set of behaviour appears in one giant commit with no explanation (perhaps copied from somewhere else)</li> a mention of a ticket number from a defunct ticket tracker with no further explanation</li> </ul> It seems to me that it doesn't take much additional effort to rattle out a sentence or two with some context on why something is being changed for the benefit of future maintainers. Especially when it's a complex patch that maybe took more than a day to create.</p> Given that each line of code is read many more times than it is written it seems that being "lazy" with explanation is not delivering the highest quality output to your client/employer/project.</p> I, like many developers, also value speed of delivery, fast iteration, early prototypes that may get rewritten; but you can still move fast while taking a few minutes for each patch to explain it. And you have to think that just because you might throw this one away, you also might not, and you just don't know yet.</p> Why good commit / pull-requests matter in commercial delivery</h2> I often run into the objection of "we don't have time for polishing git logs" in the context of commercial projects where time is money.</p> In commercial delivery teams good commits and pull requests are every bit as important. High quality commits & pull requests result in faster and more effective peer review, with less risk of missing important problems.</p> Clear intent (i.e. "why this change now") reduces review time. - Reviewers don’t have to reverse-engineer the purpose of the change, they can validate correctness quickly instead of spending time guessing.</li> Small, single-purpose commits isolate risk. - When refactors and behaviour changes are separated, reviewers can approve mechanical changes quickly and focus attention only where logic actually changed.</li> Descriptive messages enable reliable approval / sign-off. - A reviewer can confidently understand and confirm that "this change does X to solve Y".</li> Readable history accelerates future fixes. - When something later breaks or needs extension, engineers can locate the exact change and rationale quickly, shortening incident resolution and feature lead time.</li> </ul> Well-structured commits are a throughput optimization: they reduce review latency, improve the chances of catching problems early and shorten future debugging time - all of which speeds up the delivery of value rather than slowing it.</p> Pull request descriptions matter too</h2> It is common to ensure peer review by way of "pull requests". Pull request descriptions are a lot like commit messages - they are used to describe to another human the contents of change to the codebase.</p> As such the pull request descriptions perform largely the same function as commit messages, and should follow the same guidelines laid out here for the same reasons.</p> In the hierarchy of longevity and accessibility they sit between ticketing systems and git commits. Git logs will likely outlive both pull requests and tickets.</p> If you make an effort to make the git commits contain sufficient context for the reader of a patch then it's often useful to copy-paste that information into the pull request description, tidy it up for the reader and a bit more context such as screenshots.</p> How to make them better</h2> There's not much for me to add on what's already been written, so read these articles on the specifics of writing good commit messages.</p> Use the present-tense imperative</a> ("Add …" not "Adds" or "Added")</li> https://vip.wordpress.com/documentation/commit-messages/</a></li> https://robots.thoughtbot.com/5-useful-tips-for-a-better-commit-message</a></li> http://chris.beams.io/posts/git-commit/</a></li> https://gds-way.cloudapps.digital/standards/git.html</a></li> https://mislav.net/2014/02/hidden-documentation/</a></li> Pull Request Etiquette gist by mikepea</a> - covers pull request quality as well as individual commits</li> </ul> My personal additions to this list:</p> A list of highlights of changes in bullets is often nice to add, think of it as a tourist's guide to your patch. It makes it easier to spot the key changes in a large diff, and can make code-reviews more effective.</li> Hard-wrapping lines shouldn't be required, that should be an editor/display concern but unfortunately the git tooling doesn't agree so doesn't wrap anything so you might have to hard-wrap.</li> </ul> Are you expecting every tiny change to be like this?</h2> No. Some patches really don't need much explaining, e.g. re-applying default code formatting or fixing a typo; but you should always consider what context a future reader might need.</p> Are your patches atomic incremental improvements</h2> If it's hard to write a good message, it might be that you are not taking the time to craft good single-purpose commits.</p> Examples of good</h2> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f076ef44a44d02ed91543f820c14c2c7dff53716</a> (via reddit</a>) - what I like about this one is: It adds context</strong> that you could never get from code (note some is repeatedly more briefly in code comments which is a good thing)</li> It explains the new behaviour in human terms</strong>.</li> It's easy to read (good quality English prose)</li> </ul> </li> https://github.com/DFE-Digital/find-teacher-training/pull/159/commits/00e24dbc216836dd73281688491b8da355706d81</a> - what I like about this one is: It's part of a PR that is also well described & reviewed</li> It adds context (about the thing that will call the endpoint added in the patch, i.e .the reason it was created)</li> It mentions a PR in another repo that was a source for some of the code & ideas, yet more context for answering the question "why was this done and why is it like this?"</li> The co-author is attributed (github shows this which is nice), this might give you someone to talk to about if they're still around</li> It provides an outline of the patch so you don't have to parse the whole diff to get a flavour of how the patch changes behaviour. When you have a lot of patches to read because you're looking for something in the history this can be a big timesaver.</li> It gives insight into why certain decisions were made about the final shape of the patch (e.g. why just /healthcheck and not /ping as well)</li> </ul> </li> </ul> Video presentations on why good history matters</h2> Remember, your code and your commits can last a veeeeeery long time and you never know what poor soul will have to understand what you did and why years later... when you have time watch this video:</p> </iframe> </iframe> Adding ADRs</h2> Documenting broader reasons can be done with ADR</a>s (Architecture Decision Records, i.e. why we did this and not that), Y-ADRs</a> (basically highly condensed one-liner ADRs), readme files etc., ideally in ascii</a>, and even more ideally in the same commit so that a git blame</code> will lead the future reader to find the additional reasoning right there in the same patch as the change to the code.</p> Learning more about git</h2> NDC Talk: "git beyond pull & push" - Jørgen Kvalsvik - NDC TechTown 2023</a></li> </ul> Starting up a startup 2015-12-24T18:06:00.002+00:00 I've been contracting for 3 years now, which fits nicely with my drive to have real customers and build business(es). But I still have an itch. I've now found something to scratch that itch (taking the metaphor too far, sorry about that).</p> I'm going to write about my latest adventure here, from the perspective of me as a coder who's aspiring to be an entrepreneur, so if you're interested in my personal journey on this then do add my blog to your rss reader etc. of choice. I'll try and tag them all with startup so you can filter if you aren't interested in my other ramblings :-)</p> This post tells the story how I got involved in this startup, and some of the technical choices made so far and what we have in place already.</p> So here's how it started for me:</em></p> I found out via twitter that the the startup britain bus</a> was coming to Reading, so I went to say hi. They were a friendly crowd, and whilst there I found out that there's actually a new startup group in the area called ConnectTVT</a> associated with the new co-working space called grow@greenpark</a> over in green park just off M4 J11. That beats traipsing into the big smoke for me and is just the kind of melting pot I've been looking for. I know plenty of tech folks given that I'm a professional coder, but it doesn't seem sensible for me to start a business with people with the same skills and the same blindspots as me. None of my "normal" friends seem to have the same entrepreneurial itch that I do; I needed a broader pool. I went straight from the bus to grow to check it out and met some more lovely friendly people. A few weeks later (I think) I went back again for one of their startup grinds to see what was going on. There was a small group of interesting people with diverse interests and talents, from accountants to designers and a few tech folk. One of the people there, Richard</a>, had an elevator pitch for a new take on the property market which made some sense to me and seemed well received by the group. All the people there offered their various advice and help. Richard has had experience in the property market that's led him to this insight and having mulled over the idea for quite some time was interested in how this concept might become something more real. Lacking the tech background to just go and build it himself he was interested in what kind of scale of task this might be and how to find people who could help. I offered to provide advice any time and passed Richard my contact details. I didn't at that point have any plans to get further involved.</p> I very much enjoyed meeting everyone, and went home with a head full of ideas and inspiration. I found myself still thinking it all through in the small hours that night, regularly grabbing my phone to make yet another note on the trello</a> app detailing some idea about this property thing and how it could work, how you might get over the network-effect bump etc. Clearly this wasn't going away. So I messaged Louize</a> who runs the meetup to get in contact with Richard and said it had been keeping me awake with ideas, and she got us in contact again.</p> After mulling the whole thing over for a bit longer, it occurred to me that there is no reason I couldn't do more than just provide advice, and maybe I should offer to provide a more concrete involvement. After all, I think the idea is great, and it's not like I'm busy making any of my other ideas I have into businesses yet. Richard would certainly be much better able to build at a much lower financial risk with a technical co-founder. So after a load more discussions around the idea, the odd trip to the pub and some discussions around practical arrangements that is now what we are doing.</p> How's it going?</em></p> The first thing has been to really understand Richard's vision, this has mostly involved talking lots. I set up a trello board for the startup, and that's been great for doing a brain dump of all the ideas and todos, and getting them into some sort of organisation, then moving them into "do later" or "needed for launch" kind of groupings. I've also pointed Richard at balsamiq mockups</a>, which these days has a slick web based editor and presentation mode. I've used it in the past to great effect when helping stakeholders in other projects understand what they're really likely to get before all the expense of building the wrong thing. In this startup it's already been really useful for hashing out ideas of how the site might work and even for getting reactions and insight by way of putting it in front of people and seeing if they understand the concept and how you would use it if it were a real thing already. Inspiration for this process is taken from the excellent book Don't Make Me Think</a>.</p> Hosting choices...</em></p> In terms of building something real, Richard bought a domain, so we have somewhere to put it. I spent forever over technical choices</a> because basically you could</em> use just about any technology stack, any programming language, cloud PaaS, IaaS or private servers etc. and in fact they would all work. There's actually nothing to completely rule out anything. Some great things have been built on top of apparently terrible technical choices (wordpress is lovely but php that it's built with is known for being buggy and hacky). There's much stock put in "cloud" these days, and it does have benefits such as infinite scalability (of your wallet), but it comes with an overhead of complexity and potentially vendor lockin (though that like everything can be mitigated). So call me a luddite but I've gone for an old fashioned linux VPS (virtual private server) where I can easily put a ceiling on costs till we have an income. Unlike so many modern startups that rely on farming people with free shiny stuff until they have enough they can magically make money off the back of them, burning VC cash all the while in the name of profitless growth, we're going to provide users with value, and charge them appropriately for that value, which naturally puts the brakes on the kind of explosive growth of free users that can overwhelm an ambitious startup's site. So I've made a decision at last, hunted around for a provider I could trust and bought some hosting. There's a lot to choose from so I needed a way to narrow down the field, especially given there's a lot of not very good and not very well supported VPS offerings out there. I thought there might be an advert in Linux Voice</a> magazine, which is where I found my uber-geeky ISP A&A</a> (check out the boss's blog</a> if you want to know how far that rabbit hole goes), but there weren't any. So I hopped on #linuxvoice on freenode irc and asked in there, and sure enough the good folks at the mag said they'd been using bytemark</a> and had been happy with their support. Sold.</p> So I bought bytemark's basic VPS offering, and now have a server on the internet that I can point everything at and run everything we need on no matter how quirky the needs. While we have a manageable number of users this should suffice, we're not planning any multi-million pound TV adverts any time soon. I expect if we're successful we'll have growing pains no matter what platform we're using, it's non-trivial to scale even if you're on a so-called scalable platform in the first place. There's lots of gotchas, unexpected interactions and bottlenecks in any system as it grows.</p> If it sounds like I'm trying to rationalise my decision that's because I am, I'm aware that whatever technical (and even non-technical) decisions are made at this earliest of early stages have rippling effects that are hard to reverse once you build more on top of them. However equally no matter which choices you make, none of them are truly irreversible, and there are myriad ways of tackling any of the problems that come your way in time. I've had the privilege of seeing teams that are running decent scale operations in both .net and ruby on rails, which are two very different platforms with different cultures and different sets of problems, but interestingly the difference in the overall scale of the challenge and how easy it is to get things done in the long haul was not noticeably different as far as I could tell.</p> Mailing lists and landing pages:</em></p> The first goal is to have a way to let interested people know when we've got things to show them, tell them the story, and when we need their help. That means a mailing list. Bulk email isn't for the feint hearted these days in the modern world of aggressive spam filters, SPF (no not sunblock), domain keys and increasingly rigid legal constraints as we try to fight outright spam on the wild west of the internet so I don't fancy doing it all myself. My go-to service for this is mailchimp</a>, and I haven't been disappointed. But for this to work we need an email address, and that means MX records and pointing it at some mailserver somewhere. I did look into gmail as a possible shared mailbox but it doesn't seem to do what we're after, so now I have a server running anyway I can just use that. No artificial limits, no snooping, no unwanted advertising.</p> It turns out that bytemark provide a very nice customised build of debian called symbiosis</a>, which has a bunch of useful services already installed, and a watchdog service that'll restart anything that's fallen over. Nice! It didn't take me much fiddling to get the basic web hosting to behave and get an email account on it set up. I then used the domain control panel to point the A & MX records at our shiny new server (by way of a bit of chaining of CNAMES which will be slightly slower for the first lookup but will be easier to maintain). I then had to redo the web and mail hosting because I hadn't realised that bytemark had made the server do multiple domain hosting using a neat folder based system, but that didn't take too long and it's all good learning time without customers breathing down one's neck!</p> I've now done a rudimentary holding page for the domain, with a sign-up form for the mailing list, which was goal number one. You can see it here: http://www.xchain.co.uk/</a>. If you're interested in the concept or progress then please do sign up to the mailing list. It's easy to unsubscribe if you get bored of it ;-)</p> I've added google analytics</a> to the site, which is very useful albeit slightly evil, so at least we'll have some idea if it's getting any traction as we start to spread the word (taking all stats on the internet as extremely approximate of course). I might switch to one of the more libre solutions at some point but that would require a bit more time and this is after-all a commercial venture and not an open source project. Much as I like working on open source for the greater good, I haven't yet managed to make it pay the bills. I think it's important to have metrics for a startup so you can tell if what you're doing to spread the word is actually having any effect, and what might be putting off potential customers.</p> We've sent out a mailing or two to the few people we've got signed up at this early stage (hello mum!) as much for practice as anything, and I'm impressed with mailchimp's collaborative capabilities and previewing system.</p> Tech stack</em></p> You may have noticed I still haven't entirely picked a stack, nothing I have so far has locked me in to a decision. I don't want to be paying for or locked in to Microsoft's world (and the associated treadmill of change), so although I do their stuff for a living I don't want to tie my own startup to them, so that's out. I'd prefer an open source stack as I think it matters whether you enjoy working on the platform you've chosen, and open source is just so much nicer to work with as a coder / dev-ops. Currently I'm thinking a golang based server, backed with postgresql (the industry's finest database) serving up a JSON/REST API to an AngularJS + Foundation frontend. This gives us scope to rework the different parts as needed, and to bolt on dedicated mobile apps later.</p> Web design</em></p> There's no design in the current holding page as I am no designer (frustratingly, but that's a whole other career path), and we're still working out how to pull design talent into the whole shebang in a sustainable way. Richard has put me onto webflow</a>, which maybe will get us started.</p> My first learning of trying to find a freelance designer was that I need a "web designer" not just a "designer", being from the web world I hadn't appreciated the importance of the prefix and attracted all the wrong people.</p> Further reading...</em></p> Having this startup has encouraged me to broaden my reading and listening habits further. Here's some of things I've been catching up with:</p> Regret minimization vid</a> - inspiration from Bezos</li> What is code</a> - explains what I do to the non-coders</li> HanselMinutes - Startup Accelerators</a> - a timely podcast show</li> The Startup Chat podcast</a> from WPCurve</li> Last year's startup resolutions</a> via the work in startups</a> mailing list</li> Everything in LeanStack's blog</a></li> </ul> --</p> Please do drop me a line if you found this interesting in any way. tim@timwise.co.uk</a> works best for me.</p> Sql Data Viewer - preview release 2015-10-27T14:45:00+00:00 Hello dear readers,</p> I've started working on a data brower for relational databases that allows you to click through ids with foreign-keys to get around.</p> Take a look at http://www.timwise.co.uk/sdv/</a> and if you think you might be interested then please do sign up to my mailing list</a>, and let me know if you have any feedback.</p> This started as a holiday exercise in learning google's go programming language, but is intended to become a fully-fledged software product available to buy.</p> Tim</p> 100 reasons I hate ssrs 2015-08-12T19:01:00.004+00:00 Originally posted at https://timwise.blogspot.com/2015/08/100-reasons-i-hate-ssrs.html</a> which attracted some great comments.</em></p> A slightly tongue-in cheek hit-list of nasty things and vague hand-waving opinions on what makes microsoft's sql server reporting services (ssrs) such a pig to work with.</p> I don't really know of anything better so this is mostly just pointless ranting; but I'll justify it to myself by saying at least you'll know what you're getting into if you've read this before you start. SSRS seems to be more "death by one thousand paper cuts" than completely broken, so it's not so easy to say "it's shit, shalln't use it" like any good prima-donna developer would. Sorry I mean rock-star (recruiter speak). It also offers a few features that would be pretty hard to code from hand cost-effectively in something like asp.net mvc, such as user editing, multiple export formats, scheduled emails, and some of the ways you can cut-and-shut the data in the reports.</p> I make no apologies for the colourful language, it's a representation of the emotional side of having to use this heap of crap.</p> Some of these were contributed by in the comments (on blogger) after I posted the original article. (Thanks, that was a nice surprise, and it's nice to know it's not just me!)</p> http://www.allenkinsel.com/archive/2013/01/adventures-in-ssrs</a></li> doesn't bind to a port like a normal fukcing service http://blogs.devhorizon.com/reza/2008/10/20/say-goodbye-to-iis-say-hello-to-httpsys/</a></li> http://www.iis.net/learn/get-started/introduction-to-iis/introduction-to-iis-architecture#Hypertext</a></li> https://social.technet.microsoft.com/Forums/sqlserver/en-US/f2586aca-78fe-40d6-9bcd-5151bac7136f/role-of-httpsys-in-ssrs-2008-?forum=sqlreportingservices</a></li> http://blogs.technet.com/b/andrew/archive/2007/12/04/sql-server-2008-reporting-services-no-longer-depends-on-iis.aspx</a></li> http server api (aka http.sys) https://msdn.microsoft.com/en-us/library/aa364510%28VS.85%29.aspx?f=255&MSPPError=-2147217396</a></li> </ol> </li> list reservations: netsh http show urlacl</code></li> </ol> </li> </ol> </li> auth in reporting https://msdn.microsoft.com/en-us/library/ms152899.aspx</a></li> reports in VS http://curah.microsoft.com/22200/create-ssrs-reports-using-visual-studio</a></li> https://msdn.microsoft.com/en-us/library/ms173745.aspx</a></li> </ol> </li> "Explicity add new role assingment for the account you are using and check every box in sight" ~ a.n. colleague. lol</li> ignore the .rdl.data files with git. http://stackoverflow.com/questions/3424928/in-ssrs-is-there-a-way-to-disable-the-rdl-data-file-creation#3425429</a></li> </ol> </li> no folders. https://connect.microsoft.com/SQLServer/feedback/details/487106/allow-sub-folders-in-ssrs-projects</a></li> http://stackoverflow.com/questions/3309002/visual-studio-for-ssrs-2008-how-to-organize-reports-into-subfolders-in-solutio</a></li> </ol> </li> renamed a dataset, nothing fucking broke!!!!!!!!!!!!!!!!!!!!!!!!!!! even though there are reports that depend on it. On editing the report's dataset list you can see clearly "not found", but yet it still runs. what in the blazes is that all about? caching in the report editor http://stackoverflow.com/q/3424928/10245</a></li> kill the .data cache files http://stackoverflow.com/questions/3424928/in-ssrs-is-there-a-way-to-disable-the-rdl-data-file-creation</a></li> git clean -xfd</li> fuck</li> https://social.msdn.microsoft.com/Forums/sqlserver/en-US/0aa81692-352f-4c1f-a0e3-95fe6c0797ca/cachedataforpreview-in-rsreportdesignerconfig-not-honored</a></li> https://connect.microsoft.com/SQLServer/feedback/details/468482</a></li> it's the bin\</code> folder, not the .data files. Still, fuckkkk.</li> </ol> </li> to get from a report to a db you go, report > report dataset > shared dataset > db, but db is defined in the shared dataset with another name, which can be pointed to a shared data source, which is also named. and that</em> data source actually has a connection string</li> committing to tfs failed half way through because vs had locked a bunch of files I didn't even have open</li> found a param with <Value>=Microsoft.VisualBasic.Strings.Join(Parameters!Stages.Label, ", ")</Value></code> - wtf.</li> function overload matching warning wouldn't go away till I closed the sln</li> localisation is a bitch http://stackoverflow.com/questions/10953629/how-to-change-ssrs-2008-locale</a> etc</li> http://blog.ponytailbob.com/2007/10/multi-language-tips-in-ssrs.html</a></li> http://blogs.msdn.com/b/sriram_reddy1/archive/2012/01/09/localization-in-ssrs-reports.aspx</a></li> https://support.microsoft.com/en-gb/kb/919153</a></li> http://www.keepitsimpleandfast.com/2011/09/localization-of-your-ssrs-reports.html</a></li> why you no use User!Language??</li> </ol> </li> Visual Studio 2013 crashed. hard. while cancelling new report param</li> adds 00:00:00 to date fields from sql server. duuuuuuuuuuuuh (goes via .net datetime internally, but even so, not friendly)</li> changed date format, looks fine in VS, but no change in report server. wuh? deploy all</li> no auto-sizing of cols http://stackoverflow.com/questions/7851045/ssrs-tablix-column-cangrow-property-for-width</a></li> no nulls in multi-value http://www.keepitsimpleandfast.com/2012/03/how-to-pass-null-value-to-multi-value.html</a></li> need dirty hack to show "all" rather than full list http://www.mssqltips.com/sqlservertip/2844/working-with-multiselect-parameters-for-ssrs-reports/</a></li> http://stackoverflow.com/questions/12917261/optional-multi-valued-parameters-in-ssrs</a></li> http://www.bi-rootdata.com/2012/09/efficient-way-of-using-all-as-parameter.html</a></li> http://blog.ponytailbob.com/2007/10/2-shortcomings-of-multi-valued.html</a></li> </ol> </li> some fucking horror I've yet to encounter (querystrings) http://stackoverflow.com/questions/512105/passing-multiple-values-for-a-single-parameter-in-reporting-services</a></li> it has a fucking canvas size that will push over to 2 pages</li> the font kerning on a print is massively different to on web / design view</li> sorting http://stackoverflow.com/questions/9254604/why-does-my-sql-server-reporting-service-ssrs-report-appear-to-re-sort-the-d</a></li> </ol> </li> "Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index " from editing xml. thanks for the error info. fuckkkkkers</li> the ssrs gui editor is a flaky piece of shit doesn't select the right fucking textbox in the props window</li> had to restart visual fuckigjn studio</li> grrr</li> </ol> </li> the underlying xml is fucking horrific</li> <rd:Selected>true</rd:Selected></code> ----- what in the fucking blazes is that doing in there?</li> how do you deploy without connecting visual studio to production server? you fucking don't hahahahahaa</li> powerhell https://gist.github.com/timabell/7e3019bd2de802f0b259</a></li> ssbi install croaked - https://support.microsoft.com/en-us/kb/2800050?wa=wsignin1.0</a></li> [09:44:53] john doe: Tim can I have a .bak</code> file of ReportServer$MSSQL2012TempDB</code> which the stupid software seems to be unable to operate without even though it has Temp</strong> in the database name implying it will rebuild itself (at least that's what it implies to me) [09:47:48] Tim Abell: (facepalm)</li> [09:47:49] Tim Abell: sure</li> [09:48:08] Tim Abell: I did wonder, and then I thought, no they couldn't possibly need that</li> </ol> </li> dropdown doesn't work in firefox</li> no debugging http://stackoverflow.com/a/14068447/10245</a></li> Warning : The text box ‘appliedFilters’ and the image ‘urLogo’ overlap. Overlapping report items are not supported in all renderers.</li> the only options for DRY in reports suck balls http://www.3pillarglobal.com/insights/tips-tricks-ensure-consistency-sql-server-reporting-services-reports</a></li> http://harmful.cat-v.org/software/xml/</a> xml is a terrible format anyway</li> layout is in inches</li> you can change the layout to cm</li> it stores different metrics (cm/in) for each element, wtf, pick a unit</li> reflowing nicely is impossible</li> layout engine is as intelligent as a piece of paper and a pen. x,y is all you get.</li> http://www.webapplicationsuk.com/2010/07/word-html-renderer-ndash-the-road-to-hellhellip/</a></li> https://connect.microsoft.com/SQLServer/feedback/details/540183/supported-rdl-object-model-rdlom</a></li> this is the kind of bullshit that counts for helpful content on the net https://social.msdn.microsoft.com/Forums/en-US/86205ca4-13d0-4ca6-84f1-79797616f0f4/exclude-null-values-from-sum-and-avg-calculation?forum=sqlreportingservices</a> - =sum(forum_format quality_of_community)</code> = errorrrrrrrrr</li> multiple rdl xml schema in the same fucking project, completely different xml structure 2005 generated with "new report wizard" in VS 20-fucking-13: <Report xmlns="http://schemas.microsoft.com/sqlserver/reporting/2005/01/reportdefinition" xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"></code></li> 2008 <Report xmlns="http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition" xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"></code></li> 2009 from https://technet.microsoft.com/en-us/library/cc627465%28v=sql.105%29.aspx</a> - <Report xmlns:rd=http://schemas.microsoft.com/SQLServer/reporting/reportdesigner xmlns="http://schemas.microsoft.com/sqlserver/reporting/2009/01/reportdefinition"></code></li> 2010 <Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns:cl="http://schemas.microsoft.com/sqlserver/reporting/2010/01/componentdefinition" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/01/reportdefinition"></code></li> how many fucking versions??!</li> http://stackoverflow.com/questions/15539859/what-is-the-difference-between-rdl-2008-schema-and-rdl-2010-schema-feature-wise</a></li> http://stackoverflow.com/questions/9974179/is-there-a-new-version-of-rdl-schema-for-sql-server-2012-denali</a></li> http://stackoverflow.com/questions/29951653/ssrs-2008r2-visual-studio-2008-and-2008-and-2010-schemas</a> - how to not end up with old schema?!</li> https://social.msdn.microsoft.com/Forums/sqlserver/en-US/f4d14548-c592-4d8d-8185-ca683c421649/2010-schema-with-visual-studio-2010?forum=sqlreportingservices</a></li> </ol> </li> how do you upgrade a report schema? install a massive chunk of fucking sql server http://stackoverflow.com/questions/13170608/upgrade-my-rdlc-schema-from-2008-01-to-2010-01</a></li> [17:54:48] john doe: Btw have you noticed that in Print Layout view the header doesn't expand if any of the textboxes have auto-grown? [17:55:14] Tim Abell: that's because ssrs is a piece of shit from 1990 [17:55:28] Tim Abell: and it thinks A4 is the ultimate display format [17:55:47] Tim Abell: you just have to guess how much space you'll need</li> the ordering of the xml in the proj file is unstable causing diff noise</li> subreports, icky http://bhushan.extreme-advice.com/subreport-in-ssrs/</a> or nested tables http://stackoverflow.com/questions/11335655/filtering-nested-data-regions-in-ssrs</a></li> </ol> </li> some things can only be achieved with subreports, and they have to be deployed separately from the main report, meaning they can get out of sync. enjoy the fear of not knowing if you'll break something else when you upload your new version of the subreport you depend on</li> no support for "time" data type http://stackoverflow.com/questions/3846378/displaying-time-in-reporting-services-2008</a></li> The "View Report" button next to the parameters when running a report in VS is always</em> greyed-out, even though it actually works.</li> wow that's mental, hidden reports show in details view and not in tile view in the ssrs web ui</li> the report editor has a copy option for report items, but no paste, so you can't duplicate reports actually you can, but only if you know the keyboard shortcuts. 0_o - ctrl-c ctrl-v</li> </ol> </li> duplicating a report on the filesystem, and then using "add existing item" to include it puts it at the end of the list... until you rename it and then it's moved into alphabetical order causing a spurious diff. should have put in the right place in the first place. grr.</li> using the cursor keys to move textboxes around is so laggy that I overshoot every single time</li> the editor popups in visual studio are modal, so you can't refer to anything else</li> and there's no maximise button so you have to drag the fiddly border to make it bigger</li> the report editor hasn't heard of ctrl-c or ctrl-v, have to use ctrl-Ins / shift-Ins instead</li> border rendering / precedence is a fucking mess. set some borders, your report will look like a two-year-old coloured it in, and how it looks changes depending on the zoom level.</li> you have to use VB to do alternate row colours - http://stackoverflow.com/questions/44376/add-alternating-row-color-to-sql-server-reporting-services-report</a></li> if anything goes wrong with an expression all you get is "#Error". Helpful. E.g. http://stackoverflow.com/q/9144312/10245</a></li> this one time, at band camp, I edited a report definition in VS and it refused to save the new definition to disk (ctrl-shift-s, ctrl-shift-s!!). wtf. Restarted VS and all the changes were gone.</li> report editor silently adds new parameters to the report when you add new parameters to the sql. seriously. fuck off.</li> RSI-inducing UI for editing the reports. click click click clickity click</li> the sql editor has only a single undo. like ye olde notepad.</li> in the editor, you can right-click copy, you can't right-click paste. wtf. ctrl-v does paste though. wtf again. I know, I already said it, but it's reaaaaaly shit</li> the field list on a dataset is ordinal, allowing you to mismatch the select in the sql from the list of fields in the dataset and not notice</li> how do you align a textbox on the page?<</span>Top</span>></span>0.82546cm</span></</span>Top</span>></span></span> <</span>Left</span>></span>0.07309cm</span></</span>Left</span>></span></span> <</span>Height</span>></span>0.88964cm</span></</span>Height</span>></span></span> <</span>Width</span>></span>2.3025cm</span></</span>Width</span>></span></span></code></pre>fuck you!!!!</li> one goddam cell in the underlying format:<</span>TablixCell</span>></span></span> <</span>CellContents</span>></span></span> <</span>Textbox</span> Name</span>=</span>"</span>qty</span>"</span>></span></span> <</span>CanGrow</span>></span>true</span></</span>CanGrow</span>></span></span> <</span>KeepTogether</span>></span>true</span></</span>KeepTogether</span>></span></span> <</span>Paragraphs</span>></span></span> <</span>Paragraph</span>></span></span> <</span>TextRuns</span>></span></span> <</span>TextRun</span>></span></span> <</span>Value</span>></span>=Sum(Fields!qty.Value)</span></</span>Value</span>></span></span> <</span>Style</span> /></span></span> </</span>TextRun</span>></span></span> </</span>TextRuns</span>></span></span> <</span>Style</span> /></span></span> </</span>Paragraph</span>></span></span> </</span>Paragraphs</span>></span></span> <</span>rd</span>:</span>DefaultName</span>></span>qty</span></</span>rd</span>:</span>DefaultName</span>></span></span> <</span>Style</span>></span></span> <</span>Border</span>></span></span> <</span>Color</span>></span>LightGrey</span></</span>Color</span>></span></span> <</span>Style</span>></span>None</span></</span>Style</span>></span></span> </</span>Border</span>></span></span> <</span>PaddingLeft</span>></span>2pt</span></</span>PaddingLeft</span>></span></span> <</span>PaddingRight</span>></span>2pt</span></</span>PaddingRight</span>></span></span> <</span>PaddingTop</span>></span>2pt</span></</span>PaddingTop</span>></span></span> <</span>PaddingBottom</span>></span>2pt</span></</span>PaddingBottom</span>></span></span> </</span>Style</span>></span></span> </</span>Textbox</span>></span></span> </</span>CellContents</span>></span></span> </</span>TablixCell</span>></span></span></code></pre></li> Report width limited to 455in (even if I'm hiding columns using parameters against the Visibility column filter).</li> NO DYNAMIC COLUMN CREATION (ridiculous!)</li> Selection of multiple columns and setting attributes is ridiculously flaky. This is because I wanted to reduce the column width and font to comply with Point 71 (max report width)!!</li> We have to restart Reporting Server services frequently or our charts won't show up. Eh?</li> Cut and paste columns? Nope!</li> Disappearing "Report Data" menu: http://stackoverflow.com/a/28883272/1024</a></li> SSDT for VS2015 upgrades reports to 2016 schema as soon as they're opened (WAT?!) and ignores the TargetServerVersion being set to <=2014 http://stackoverflow.com/questions/37816216/deploy-of-a-report-with-ssdt-2016-generates-error</a> what if we haven't upgraded our production server, hmmm?</li> Okay so when you run a build, VS2015 then downgrades</strong> the rdl to the right schema to match the project's target server version before putting it in the build folder. What could possssssibly go wrong. 0_o Complexity++</li> Install SSRS in only 41 easy steps https://thecodeattic.wordpress.com/category/ssrs/</a></li> Build > "Skipping 'SomeReport.rdl'. Item is up to date." - No it isn't, I've deleted it from the friggin bin folder. So clearly it has some stupid cache of what's it thinks is on disk rather than, oh I don't know fucking checking the disk. Sheesh. If it's that optimized why is everything still so damn slow?!</li> Building a project with lots of reports is slow. Even if nothing changed.</li> Intermittently get "[rsInvalidReportDefinition] The definition of this report is not valid or supported by this version of Reporting Services. The report definition may have been created with a later version of Reporting Services, or contain content that is not well-formed or not valid based on Reporting Services schemas. Details: Data at the root level is invalid."</em> - I am not alone. https://connect.microsoft.com/SQLServer/feedback/details/2988044/randomly-get-rsinvalidreportdefinition-when-previewing-report</a> - A rebuild fixes it for me, for a while at least.</li> Sometimes when you edit an embedded dataset it completely fails to persist any of your changes to disk. And when you close the report they are lost. Handy. Thanks fuck for git.</li> Assumes you've never heard of source control and creates numbered backups of report files (notably on auto-upgrade). Also has messages like "delete will permanently delete this thing</em>" - no it won't I have source control; I wonder if the SSRS know what that is.</li> I've never managed to crash VS2015 so many times in one day. Omg you didn't click there did you? I wasn't ready! crash</em> ... again</em></li> Change the files on disk and the preview window often doesn't notice</li> Preview window silently fires a 'build' of the reports. Sometimes.</li> Generates broken Shared-datasets http://stackoverflow.com/a/38753141/10245</a></li> If you break the xml of a shared-dataset the entire project</em> will fail to load. w-t-f.</li> Renaming datasets etc just breaks everything rather than updating references.</li> The .rptproj</code> file has a <state></code> tag at the top which is base64 encoded xml (WAT?! xml in xml. eerrrrr), which is information about source control (arse-about-face or what! source code that controls the source-control</em> [that controls the source, that controls the source-control that controls the source...]; even the words are circular!!</em>). In the base64 you'll find a <SourceControlInfo></code> tag.<</span>Project</span> xmlns</span>:</span>xsi</span>=</span>"</span>http://www.w3.org/2001/XMLSchema-instance</span>"</span> xmlns</span>:</span>xsd</span>=</span>"</span>http://www.w3.org/2001/XMLSchema</span>"</span> ToolsVersion</span>=</span>"</span>2.0</span>"</span>></span></span> <</span>State</span>></span>$base64$PFNvdXJjZUNvbnRyb2xJbmZvIHhtbG5zOnhzZD0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEiIHhtbG5zOnhzaT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEtaW5zdGFuY2UiIHhtbG5zOmRkbDI9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDAzL2VuZ2luZS8yIiB4bWxuczpkZGwyXzI9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDAzL2VuZ2luZS8yLzIiIHhtbG5zOmRkbDEwMF8xMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDA4L2VuZ2luZS8xMDAvMTAwIiB4bWxuczpkZGwyMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEwL2VuZ2luZS8yMDAiIHhtbG5zOmRkbDIwMF8yMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEwL2VuZ2luZS8yMDAvMjAwIiB4bWxuczpkZGwzMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDExL2VuZ2luZS8zMDAiIHhtbG5zOmRkbDMwMF8zMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDExL2VuZ2luZS8zMDAvMzAwIiB4bWxuczpkZGw0MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEyL2VuZ2luZS80MDAiIHhtbG5zOmRkbDQwMF80MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEyL2VuZ2luZS80MDAvNDAwIiB4bWxuczpkZGw1MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEzL2VuZ2luZS81MDAiIHhtbG5zOmRkbDUwMF81MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEzL2VuZ2luZS81MDAvNTAwIiB4bWxuczpkd2Q9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vRGF0YVdhcmVob3VzZS9EZXNpZ25lci8xLjAiPg0KICA8RW5hYmxlZD50cnVlPC9FbmFibGVkPg0KICA8UHJvamVjdE5hbWU+U0FLPC9Qcm9qZWN0TmFtZT4NCiAgPEF1eFBhdGg+U0FLPC9BdXhQYXRoPg0KICA8TG9jYWxQYXRoPlNBSzwvTG9jYWxQYXRoPg0KICA8UHJvdmlkZXI+U0FLPC9Qcm92aWRlcj4NCjwvU291cmNlQ29udHJvbEluZm8+</span></</span>State</span>></span></span> ...</span></span></code></pre>decoded "state" contents:<</span>SourceControlInfo</span> xmlns</span>:</span>xsd</span>=</span>"</span>http://www.w3.org/2001/XMLSchema</span>"</span> xmlns</span>:</span>xsi</span>=</span>"</span>http://www.w3.org/2001/XMLSchema-instance</span>"</span> xmlns</span>:</span>ddl2</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2003/engine/2</span>"</span> xmlns</span>:</span>ddl2_2</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2003/engine/2/2</span>"</span> xmlns</span>:</span>ddl100_100</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2008/engine/100/100</span>"</span> xmlns</span>:</span>ddl200</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2010/engine/200</span>"</span> xmlns</span>:</span>ddl200_200</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2010/engine/200/200</span>"</span> xmlns</span>:</span>ddl300</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2011/engine/300</span>"</span> xmlns</span>:</span>ddl300_300</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2011/engine/300/300</span>"</span> xmlns</span>:</span>ddl400</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2012/engine/400</span>"</span> xmlns</span>:</span>ddl400_400</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2012/engine/400/400</span>"</span> xmlns</span>:</span>ddl500</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2013/engine/500</span>"</span> xmlns</span>:</span>ddl500_500</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2013/engine/500/500</span>"</span> xmlns</span>:</span>dwd</span>=</span>"</span>http://schemas.microsoft.com/DataWarehouse/Designer/1.0</span>"</span>></span></span> <</span>Enabled</span>></span>true</span></</span>Enabled</span>></span></span> <</span>ProjectName</span>></span>SAK</span></</span>ProjectName</span>></span></span> <</span>AuxPath</span>></span>SAK</span></</span>AuxPath</span>></span></span> <</span>LocalPath</span>></span>SAK</span></</span>LocalPath</span>></span></span> <</span>Provider</span>></span>SAK</span></</span>Provider</span>></span></span> </</span>SourceControlInfo</span>></span></span></code></pre>Enough xml namespaces for you?</li> The state base64 changes all the time causing diff noise.</li> </ol> I know I can't count, but if ssrs is going to make so little effort I don't see why I should. And to be honest "100" seemed a lot more like comic exaggeration when I titled my tomboy note which only had 5 or 6 grumblings in it, I wasn't actually expecting to get within spitting distance of the original number!</p> Doesn't mean I won't use it again mind, just don't promise to like it.</p> If you liked this, you might also like the ssrs deployment tool I sometimes look after: https://github.com/timabell/ssrs-powershell-deploy</a> (mostly not my work, just pulled together a bunch of contributions).</p> LibreOffice mail merge - "data source 'SOURCE' was not found" 2015-01-12T22:31:00.002+00:00 So another year on, LibreOffice 4.2.7.2 (via Linux Mint 17.1) still has a dog's breakfast of a mail merge feature, hey ho, hopefully it might actually get fixed following the fork from OpenOffice and the change in contribution methods.</p> Anyways</p> So I've moved machines, copied my files across and for some reason my mail merge has soiled itself and now bleats "data source 'SOURCE' was not found"</em> which is as unhelpful as it is infuriating, especially given that the "check connections" button is exactly the wrong place to look for an answer.</p> Turns out you actually get this if even just a single field in your document is 'broken'. How do you tell which ones are broken? Well you have to change them all to just be sure. Sigh.</p> The fix for me today was as follows (though with such a messy feature there's unlimited ways it can break):</p> Hit F4 and check that your connection to the spreadsheet actually exists and works, and unbreak anything you find therein. While you're in there you can marvel at how it requires a whole other file (.odb) just to remember how to get to a spreadsheet. (See below for fixing this)</li> Turn on the field names so you can see what the f*** is actually going on with "View > Field Names (ctrl+f9)</em>" which will show you the fully qualified field name, which might even be completely wrong. You can now see that for whatever reason (insanity?) it embeds more than just the field name at the field place-holder.</li> And finally the way you actually fix the broken fields it's failing to tell you about actually lies under the menu item "Edit > Fields</em>", where you can change all the broken references one at a time to the correct place.</li> For bonus points, if it the field looks right but is silently broken somehow then you have to change the field to something else, hit okay, and then change it back again for anything to actually change, which is annoying if you have a lot of fields.</li> </ol> Fragile much?</p> Another fix I've just discovered is you can rename your data source to match the name defined in the fields (assuming they're all the same) and it'll start working again.</p> Fixing the .odb file</h4> If you're stuck on point 1, here's how you fix it, also completely non-obvious and full of apparent dead-ends and dubious information.</p> Give up on trying to do this in writer, it doesn't seem possible, in spite of false hope from the data sources tool, it only allows you to select .odb (database) files, not spreadsheets.</li> Open up "libreoffice base", which pops open the database wizard</li> Choose "connect to an existing database"</li> In the dropdown choose "Spreadsheet"</li> Next</li> Browse for your spreadsheet</li> Next</li> Leave "register database for me" selected</li> Leave "open the database for editing" checked</li> Finish</li> It prompts to save the new database (.odb), I suggest saving it in the same folder as the spreadsheet to save future confusion.</li> You now have the database open in "base", you should see your spreadsheet sheets listed as tables</li> Open a table (i.e. a sheet) and check you can see the spreadsheet contents</li> Close "base", saving changes</li> Return to your writer document</li> Open the data sources again (F4), you should now be able to browse your spreadsheet via your newly created database.</li> </ol> Simpler than getting planning permission out of a vogon. :-/</p> Hope that helps some other poor open source die-hard who has work to do.</p> Useful refs:</p> http://davidmburke.com/2011/08/10/mail-merge-in-libreoffice/</a></li> https://forum.openoffice.org/en/forum/viewtopic.php?f=30&t=29708</a></li> https://www.libreoffice.org/bugzilla/buglist.cgi?quicksearch=mailmerge</a></li> http://ask.libreoffice.org/en/question/19590/can-you-import-a-libreoffice-calc-spreadsheet-into-a-libreoffice-database/</a></li> </ul> throw vs throw ex vs wrap and throw in c-sharp 2014-05-10T10:23:00+00:00 I've come across the throw</code> vs throw ex</code> 'debate' a few times, even as an interview question, and it's always bugged me because it's never something I've worried about in my own c# code.</p> Background</h1> So here's a typical example of the throw vs throw ex thing: https://stackoverflow.com/questions/730250/is-there-a-difference-between-throw-and-throw-ex</a></p> Basically it revolves around either messing up the line numbers in your stack trace (throw ex;</code>) or losing a chunk of your stack entirely (throw;</code>) - exception1 and 2 respectively in this nice clear answer: http://stackoverflow.com/a/776756/10245</a></p> </a> </div> The third option</h1> I've just figured out why it's never been an issue for me.</p> Because in my own code, whenever I catch and re-throw I always</em> wrap another exception to add more context before rethrowing, and this means you don't have either of the above problems. For example:</p> private static void ThrowException3() {</span></span> try {</span></span> DivByZero(); // line 43</span></span> } catch (Exception ex) {</span></span> throw new Exception("doh", ex); // line 45</span></span> }</span></span> }</span></span></code></pre> Exception 3:</p> System.Exception: doh ---> System.DivideByZeroException: Division by zero</span></span> at puke.DivByZero () [0x00002] in /home/tim/repo/puker/puke.cs:51</span></span> at puke.ThrowException3 () [0x00000] in /home/tim/repo/puker/puke.cs:43</span></span> --- End of inner exception stack trace ---</span></span> at puke.ThrowException3 () [0x0000b] in /home/tim/repo/puker/puke.cs:45</span></span> at puke.Main (System.String[] args) [0x00040] in /home/tim/repo/puker/puke.cs:18</span></span></code></pre> Obviously 'doh' would be something meaningful about the state of that function ThrowException3()</code> in the real world.</p> Full example with output at https://gist.github.com/timabell/78610f588961bd0a0b95</a></p> This makes life much easier when tracking down bugs / state problems later on. Particularly if you string.Format()</code> the new message and add some useful state info.</p> Why publish open source when you are commercial? 2014-03-08T19:07:00+00:00 Why open source your commercial projects?</p> Forces you to decouple them from other internal systems.</li> Encourages thinking in terms of reusable modules, which is better for internal reuse just as much as public reuse.</li> Possibility of contributions to systems useful to your business by others.</li> Easier reuse within your organisation (the public internet is a better search and sharing system than any internal systems).</li> Reputation advantages, the best coders often like to work in open and forward-thinking companies, and having public shared code is a great sign of such an organisation.</li> </ul> Do it early</p> Preferably push your very first commit straight to github.</li> Do it before it has a chance to be tightly coupled to internal systems, otherwise you'll have to unpick it and it will be less decoupled from day one, and inertia might mean that in spite of the best intentions you then never publish it.</li> You'll have it in mind that every commit is public from day one, avoiding adding internal config etc and forcing you to factor it out into config which is all round a good thing.</li> Don't wait for your code to be perfect, there are compromises in all code and sharing something imperfect is better than sharing nothing.</li> </ul> </a> </div> Worried about the brand?</p> Commit under personal email addresses and push to personal github accounts. You can always setup a corporate github account later when you are feeling more confident.</li> </ul> Of course I'm not saying you should open source everything, for example your core product's codebase should probably not go on github if you are a product company!</p> Be brave, be open.</em></p> Props to Tom Loosemoore</a></p> Bash command line editing cheat sheet 2014-02-10T18:33:00+00:00 https://www.gnu.org/software/bash/manual/bashref.html#Readline-Interaction</a></p> ctrl-a/e</strong> start/end of line</p> </li> alt-f/b</strong> forward/back a word</p> </li> ctrl-w/alt-d</strong> delete to start/end of word</p> </li> alt-.</strong> add argument from previous command (repeat to cycle) - love this one</p> </li> ctrl-shift-_</strong> undo (i.e. ctrl-underscore)</p> </li> ctrl-y</strong> paste (yank) deleted text</p> alt-y</strong> paste older deleted text instead</li> </ul> </li> prefix with alt+digit</strong> (0-9) to do multiple, e.g. delete two words</p> </li> start with alt-minus</strong> to go backwards</p> </li> </ul> Just a few notes I threw together for my own benefit. I finally got around to learning a bit more about editing commands on the Linux shell / terminal.</p> </a> </div> Converting kml to gpx with python 2014-02-03T20:27:00+00:00 Today I wanted to geo-code some of my photos.</p> I have an SLR digital camera (no gps of course), and an android phone. I recorded a track with My Tracks</a> from google on the phone. (Not entirely recommended but works). I then fired up digikam to run the geo-correlation and add lat-long to the exif of the files only to discover digikam doesn't know how to read kml</a>. Fooey.</p> </a> </div> I looked to gpsbabel, but it apparently can't handle this style of kml file, as differentiated by the coordinates being in the following style of markup:</p> <gx:Track></span></span> <when>2014-01-25T18:00:13.955Z</when></span></span> <gx:coord>-1.885348 50.769434</gx:coord></span></span> <when>2014-01-25T18:00:14.565Z</when></span></span> <gx:coord>-1.885193 50.769328 53.20000076293945</gx:coord></span></span> <when>2014-01-25T18:00:58.566Z</when></span></span></code></pre> So I wrote a python script to munge it into gpx shape:</p>

Upgrading indirect NuGet dependencies

2022-02-15T00:00:00+00:00

It seems microsoft have produced a solution to this problem, along with the long awaited solution-level package management. Read all about it here: https://devblogs.microsoft.com/nuget/introducing-central-package-management/#transitive-pinning</a></p>

If you want to convert to central package management this tool works a charm: https://github.com/Webreaper/CentralisedPackageConverter</a> (via reddit</a>)</p>

About</h2>
The state of the art for dependency management in dotnet land. Having used ruby bundler and npm this makes me cry.</p>
I really hope I'm massively wrong about the following information. Please do tell me if I'm wrong and NuGet dependency management is not actually as bad as this!</p>

Paket, a NuGet alternative</h2>
If you have the option then look into replacing nuget with paket</a> for your projects as I gather this has the below problems solved.</p>

Indirect dependencies</h2>
Say you depend on Microsoft.AspNetCore.Http v2.2.2</a> (which is the latest available at time of writing), which in turn depends on Microsoft.AspNetCore.Http.Abstractions >= v2.2.0</a> (also currently the latest) which in turn depends on System.Text.Encodings.Web >= 4.5.0</a></p>
And then you discover, say, just for a fun example, that System.Text.Encodings.Web v4.5.0 has the Remote Code Execution (RCE) Vulnerability CVE-2021-26701</a>.</p>
Thankfully Microsoft have released a patch release with a fix for the CVE: System.Text.Encodings.Web v4.5.1</a></p>
So obviously you'll just upgrade your packages, run CI, make a cup of tea and ship to prod, right? Wrong.</p>
But it says `>= 4.5.0</code>, so shurley it'd just upgrade it?...</p>`
`Apparently not. It seems that for whatever reason nuget actually prefers the lowest</em> compatible patch release. Go figure. So it won't upgrade.</p>`

Checking the source of truth</h3>
Because I'm starting to doubt my sanity at this point, let's dive in to the source code for the middle package to make sure we're not just being ignorant about how it actually works.</p>

Microsoft.AspNetCore.Http.Abstractions</code> has abstracted out the version info for its dependencies to another file, you can see the defined version of the ...Encodings.Web dependency as at v2.2.0 here</a></p>
<SystemTextEncodingsWebPackageVersion>4.5.0</SystemTextEncodingsWebPackageVersion></span></span></code></pre>
which is then referenced in the .csproj</code> here</a></p>
<PackageReference Include="System.Text.Encodings.Web" Version="$(SystemTextEncodingsWebPackageVersion)" /></span></span></code></pre>
On the face of it you'd think that this meant it was pinned to an exact version, but microsoft in their infinite desire to pander to the lowest common denominator (i.e. idiots and n00bs) decided that 1.0</code> should actually mean 1.*</code>.</p>
Notation</th> Applied rule</th> Description</th></tr></thead>

1.0</td> x ≥ 1.0</td> Minimum version, inclusive</td></tr>
...</td> </td> </td></tr>
</tbody></table>
Source: NuGet version range definitions</a></p>
It doesn't even define whether it will upgrade major, minor, or just patch releases. Horror-show.</p>
lock files</h2>
Lock files don't actually help much here, but are a good idea, so if you want to use them here's what I've learned about packages.lock.json</code> files in Microsoft-land.</p>
NuGet is veeeeeeeeeeery late to the lockfile game. A lockfile is just a second file that lists the dependency tree that was actually calculated for a given set of top level dependencies. Useful for repeatable builds and lets you see what's being resolved, but doesn't actually help fix the problem here as the point of lockfiles isn't to manually futz with them, it's just to do repeatable builds. And futzing with them would confuse everyone, and is made a little harder with the inclusion of checksums.</p>
It is possible to manually enable lockfiles</a> by adding the following to a csproj file:</p>
<</span>PropertyGroup</span>></span></span>
    <</span>RestorePackagesWithLockFile</span>></span>true</span></</span>RestorePackagesWithLockFile</span>></span></span>
</</span>PropertyGroup</span>></span></span></code></pre>
I very much enjoyed this multi-comment rant at the NuGet team</a> for basically being behind the times and always making poor decisions.</p>
There is a slightly harder problem that microsoft have so far failed to sort out of managing packages across a whole solution</a> (i.e. multiple projects). Though I can't help thinking that it's mostly hard because of the mess they've allowed to build up so far. (For example encouraging people to create multiple solutions with a mix of project files included, and the horror that is the one-giant-tree TFS system.)</p>
Forcing the upgrade</h2>
So it seems the only way to force the upgrade of the vulnerable "transient" dependency (i.e. one not directly specified in your .csproj</code> file) is to add it as an explicit dependency directly to the .csproj</code> PackageReference</code> entries. This is lame, and likely to cause confusion later. Make sure your commit message is clear as to why you've added this seemingly nonsense dependency.</p>
<</span>ItemGroup</span>></span></span>
	<</span>PackageReference</span> Include</span>=</span>"</span>System.Text.Encodings.Web </span>"</span> Version</span>=</span>"</span>4.5.1</span>"</span>/></span></span>
</</span>ItemGroup</span>></span></span></code></pre>Getting notified of dependency updates</h2>
If you run any production code you can run an analysis tool like Snyk.io</a> or dependabot</a> to check for such CVEs popping up in your dependency graph.</p>
You'll probably want to install a local version of whatever you use to avoid the CI loop for every change you make, snyk has a visual studio extension</a>.</p>


Phone setup notes
2021-12-17T00:00:00+00:00

About</h2>
Well this makes a good counterpart to the laptop setup post</a>. Similar idea, it's just a place for me to keep track of all the things involved in building a new phone from scratch. This one also explains to people who look at me like I'm mad why I don't just "do what everyone else does".</p>
Why don't you just...?</h2>
Most people don't think their phone install deserves a blog post, or even a lot of thought beyond a visit to the Apple "genius" bar (lol) so they can push the "copy everything to my new phone button". I'm not most people. I've been running Linux since approximately Ubuntu 5.04 Hoary Hedgehog</a> in 2005 (the brown one). Even while being a ".NET developer", back when Microsoft were still trying to simultaneously ignore and destroy Linux</a>. What strange times we live in now with dotnet being open source and cross-platform, has the leopard changed its spots?</p>
There are many factors in this battle for the phone in your pocket.</p>

The GPL's fundamental truth is that there is a tension for control over your devices between the creators of software and the users of that software. iOS and Android both believe that the users can't be trusted with full control over their device because "security". I believe this balance of power is important. Too far in either direction and things start to go badly:

All the power in the hands of vendors at the expense of users results in exploitation, unfair pricing, anti-competitive practices, monopolies, duopolies (google+apple), cartels, and lack of innovation and quality due to lack of competition.</li>
On the flip side all the power in the hands of "users" (or at least user-developers) as per the pure GPL and it becomes difficult or impossible for capitalist processes to fund innovation and engineering effort due to the inability to capture generated value.</li>
</ul>
</li>
The apps matter, I do actually want to use a phone for useful things, and increasingly web applications are a poor cousin to their equivalent apps. (Facebook, horror that it is, oscillates between making a browser unusable and making it just about possible to read and interact, they really</em> want all that juicy extra data that you can get from being an installed app).</li>
</ul>

iOS and Android both believe that users can't be trusted with full control over their device</p>
</blockquote>
This time in smartphones is a moment like my first linux install around the year 2006. I hated the main operating system (Windows) that I was locked in to. All the app[lication]s that I relied on (outlook, visual studio etc) were locked to the operating system I hated. And now in 2021 all the proprietary apps on phones are locked to android / iOS (gmail, authenticator, waze, google maps, banking apps, whatsapp etc.)</p>
I could just install OpenGApps or similar to get the google play store on top of a slightly more free android variant; then install all the proprietary apps again, or I could see just how much I can do without google services by using microG and open source alternative apps like k9mail and maps.me. I've already started this journey by trying out many free alternatives and think it's worth pushing ahead. Running a build with no google services will be a good acid-test of how far I've got even if I have to go running back to the proprietary google services.</p>
In my switch to Linux from Windows I just tried to do as much as I could with free software such as the gimp and thunderbird, often still installed on Windows, and then bit by bit I needed to fire up Windows less and less often as I found alternative apps and ways of doing things.</p>
Any progress I can make here will make the jump to a fully free phone OS less painful in the future, and I'm increasingly confident that day will come just like I don't think twice about running Linux Mint even for work these days.</p>
Nursing Homes and Neighbourhoods</h3>
I highly recommend reading this article - Purism, The Future of Computers: The Neighborhood and The Nursing Home</a>. It wonderfully illustrates the binary choice between the apple/google duopoly and the DIY open source and privacy movement with the rather apt analogy of nursing homes (apple etc where your needs are met but you give up control of your environment) and neighbourhoods (open source where it's up to you not let the wrong people in to renovate your house).</p>
Why not just Apple iOS?</h3>
Yes I know the iPhone crowd can just push a button on the new phone and have it look like the old phone, but iOS is the polar opposite of what I want my technology to be: under my</em> control. That means root. That means open source. That means GPL.</p>
iOS certainly does have some unique advantages that I will be sad to miss out on but not sad enough to give up all the other things I care about.</p>
iOS, the good</h4>

A secure boot chain, remarkably hacker-proof - even with physical device access which is very impressive.</li>
Impressively effective device-to-device migration of user settings and data (android failed miserably at this in my experiments)</li>
</ul>
iOS, the dubious-to-bad</h4>

Complete reliance on a single vendor for looking after all your precious "data". This in my view is risky no matter how good the vendor is.

In 2012 this was demonstrated as Mat Honan had all</em> his data and devices wiped and locked remotely when a hacker took over his Apple account</a>, and he had no other copy of all his precious family photos. A cautionary tale for all of us.</li>
Someone I know lost everything on their iPhone when the OS upgrade process somehow managed to do the following:

backup the phone to the iCloud</li>
wipe the phone and install the updated iOS (I understand this is normal procedure)</li>
back up the blank phone to iCloud overwriting</em> the only backup (apparently there was only a single slot)</li>
restore the blank backup to the updated phone</li>
declare success, leaving my friend with all his data gone and a factory-installed OS</li>
</ol>
</li>
</ul>
</li>
No root (a problem for syncthing, independent backups and general user-control)</li>
No user-replaceable OS (aka ROM)</li>
No open source</li>
All the apps cost more!</li>
</ul>
Why not just stock android?</h3>
"So you must be an android dude then" you say. Well no because frankly android isn't much better these days. Ever heard of AOSP? (The Android Open Source Project)</a> Well that's an ever-shrinking piece of what people call Android these days. Piece by piece google have been replacing open-source android with proprietary rewrites</a>. Combine that with proprietary drivers for the endless churn in hardware. If you can even install ASOP on a device good luck getting much to work. The idea of android being "the open source one" in the true spirit of the GPL is a distant and fading memory.</p>
There is a cartel of industry players called the Open Handset Alliance</a> http://www.openhandsetalliance.com/index.html</a>. These industry groups can be good but I think in this case they are a barrier to innovation and openness.</p>

"OHA members are contractually forbidden from producing devices that are based on competing forks of Android"

~ Wikipedia on OHA</a></p>
</blockquote>
Ars Technica: "Google’s iron grip on Android: Controlling open source by any means necessary"</a></p>
Root and backups</h4>
One of the big drivers I've had for getting a non-standard phone setup is that I don't want to rely on on google to backup everything on my phone in case it dies, but because Android is by default locked in by secure boot and doesn't give the user any root access or ability to get root access that's a bit of a problem for the normal method of backing up a device.</p>
A laptop backup is a question of backing up /home</code> plus a dotmatrix/bootstrap</a> setup for regaining interesting configuration etc.</p>
Android is a bit more complex because it has a couple of storage areas:</p>

an [emulated] 'sdcard' this is kinda like /home</code> on a laptop and has things like downloaded files; some of the apps use this to save things to</li>
storage for every individual app (I learned about the existence of this by losing all my app data)</li>
</ul>
Every app on android runs under a separate userId, which allows Android to ensure that apps can't read each other's data. This is a good thing for security, but that means if you install a "backup app" it can't read any of the data you actually want backed up. And it can't get root because that's not available to you as a user or to any non-system apps.</p>
So that means to backup independently of google, you need root, and to get root you need to unlock the whole boot chain.</p>
I did have root on my OnePlus 5t, but an operating system update helpfully removed that for me... breaking my backups... and root access couldn't be restored without wiping the phone... including the data I didn't have backed up because it had broken my backups. Gah. Off the back of that experience I want to make sure that I'm not set up to fight some bloody vendor that doesn't believe I should have root.</p>
Any hope of an effective root is dwindling. According to https://www.androidpolice.com/a-new-method-for-hiding-root-in-magisk-may-soon-emerge/</a> the developer of Magisk Hide that prevented root being detected and thus allowed shitty apps to continue to be run has been hired by google</em>. That's google following the advice of "keep friends close, keep your enemies closer" close if ever I've seen it. No better way to shut the project down than to pay off the developer with a "salary". Cough.</p>
Something other than iOS or Android and its derivatives?</h3>
It's all about the apps.</p>

An operating system is not that useful on its own.</p>
</blockquote>
There's a reason Ballmer shouted "developers developers developers"</a> (amusing as this meme has become in the abstract). An operating system is not that useful on its own. Operating systems live and die by the applications (aka apps) available on them, and the utility and value they in turn bring to their users.</p>
iOS and Android have utterly captured the app market on phones, creating a duopoly in the phone space.</p>
Even the mighty Nokia (with Symbian</a>), Research In Motion (RIM) with Blackberry</a>*</code> and Microsoft with Windows Phone</a> failed to win against them and eventually threw in the towel.</p>
*</code> Blackberry seems to have all but fallen off the radar, they started shipping android in the end basically being a android phone with a physical keyboard thus becoming part of the apple-google duopoloy..</p>
So for me that rules out for the time being all the Linux based phones such as a Librem 5 running PureOS</a>. (See below).</p>

It's only when alternatives like Librem/PureOS gain traction that we'll see the makers of apps, websites and phones have to create and adopt open standards that allow further diversity and innovation.</p>
</blockquote>
The fact the Librem exists and seems to be sustainable are encouraging signs for the breaking of the google+apple duopoly on mobiles. It's only when alternatives like Librem/PureOS gain traction that we'll see the makers of apps, websites and phones have to create and adopt open standards that allow further diversity and innovation. Currently we are seeing a repeat of the "works best in Netscape / Internet Explorer" duopoly of the browser wars that was eventually broken forever by Firefox, Opera and friends forcing websites to stop using proprietary extensions and stick to open standards.</p>
Dearly departed</h4>
Nokia / Symbian</h5>
Symbian was briefly ahead on features as an OS in my opinion. I had a Sony Ericsson P800</a> and P910</a> and in their day they were almost</em> amazing phones, hampered by lacking hardware (2g at the dawn of 3g, serial port connectivity emulated over usb, and for god knows what reason Sony were still trying to beat SD cards with memory stick</a>). Consumers flocked away from them in droves to the new iPhone.</p>
Microsoft / Windows Phone OS</h5>
Microsoft's "Windows Phone" was a triumph of a phone operating system</a>, and everyone I knew that had one absolutely loved the operating system. But Microsoft just couldn't persuade the big-name mobile-app developers</a> (i.e. companies) to invest the endless effort required to create and maintain a third version of their apps for another app store.</p>
OpenMoko / Neo FreeRunner</h5>
I had high hopes for the truly open Neo running OpenMoko, but it didn't stand a hope in hell amongst the rapidly evolving tech and software at the time. I bought the Neo FreeRunner</a> and it was an exciting time, but it quickly became a relic.</p>
CyanogenMod</h5>
After many years of being the goto android-variant mobile OS for non-conformists, sadly CyanogenMod</a> imploded in 2016. The code lives on in LineageOS (see below).</p>
Promising futures</h4>
Two currently active non-android physical phone projects, and several more ongoing Linux based (i.e. non-android) mobile-friendly operating system efforts. Although these will never cross the radar of most people just like Linux didn't for many years) these are important foundational projects.</p>
Purism's Librem 5 phone hardware running PureOS (Linux)</h5>

Librem 5 phone</a></li>
PureOS</a></li>
</ul>
The Librem 5</a> is worth a look if you want to take it to the next level of openness, however it comes with a compromise in specifications, a hefty price tag (inevitable due to lower production volumes) and no android / iOS apps.</p>
You might however want to consider spending your dollars as a vote for the future you want to see.</p>
The Librem 5 runs Linux (PureOS), not Android</a>. This is great but as of 2021 does mean it suffers from the "app" problem (see above).</p>
Because of the "app" problem I'm not ready to make the jump to a Linux based smartphone that can't run android apps. I'm also not particularly keen to drop quite this much in specification for the benefit.</p>
Personally I'm happy with the OnePlus compromise of specification plus apps versus openness for now, but I'm very glad to see Purism pushing the frontier of openness and maybe I'll join their users one day.</p>
PinePhone Pro phone hardware (Linux, for now)</h5>
The PinePhone Pro</a> is a low spec but very open Linux phone. A follow up to the original PinePhone.</p>
"pre-orders from developers on October 15, 2021, and expect to have them delivered by December"</p>
Phones like this are important in building momentum in the capability of non iOS/Android software even though they will never be mainstream phones themselves.</p>
This is a Linux phone, though in theory GloDroid</a> will provide a build of Android (AOSP) for it, but don't hold your breath. - OmgUbuntu article on PinePhone+GloDroid</a>.</p>
UBPorts Ubuntu Phone (Linux)</h5>
Canonical killed the Ubuntu Phone and the Ubuntu Touch</a> operating system, but the software lives on in ubports</a>. (Presumably that's short for UBuntu Ports)</p>
Plasma (Linux KDE for phones)</h5>
Plasma Mobile</a> is a variant of the KDE Plamsa desktop specifically for mobiles.</p>
Learn more about Plasma on Wikipedia</a></p>
Mobian - (Debian Linux for phones)</h5>
Mobian</a> is encouraging to see. Debian Linux's strenth has always been its governance and stability. Debian has been a rock in the stormy seas of open source even as different people have passed through its team. This is because debian has a strong set of organisational rules that are above and beyond any individual contributor. The fact that they are working on a mobile variant is very encouraging.</p>
PostmarketOS (Linux)</h5>
PostmarketOS</a> is an Alpine Linux (i.e. very small) variation designed to breath life into otherwise obsolete hardware. I expect this will become more useful as phone hardware stops becoming utterly obsolete due to major changes in basic expectations and environment.</p>
Secure boot</h2>
There's the secure boot chain and device locking. This can be a good thing or a bad thing depending on the power dynamic between users and providers (and maybe even the state/government).</p>
It is true that your device is less "secure" with the bootloader unlocked (as the phone likes to scream at you every time you boot it up), at least to attackers with physical access to your device, however I think that's worth the trade for full control of your device (that you paid for and "own").</p>
Security is not an absolute</a>. You have to consider cost, inconvenience, human behaviour, the value of an attack target, the motivations of different actors (state, organised crime, hackers, opportunist thieves, trolling friends etc). For me an encrypted data partition and a passcode will do for physical security. In theory you can get full secure boot chain for custom operating systems but even on a Linux laptop this is non-trivial currently.</p>
There's an important difference here between google/apple considering the security of their entire device eco-system in the hands of every kind of user, and the security of an individual's device customised to suit them. If google turn off secure boot for everyone then it makes it that much more likely that someone will successfully develop and spread an android virus that corrupts the boot chain. If I turn it of it's a rounding error on the probability, and a marginal difference to the "security" of my own device.</p>
Apple's secure boot chain</h3>
Apple don't want you to fiddle with the OS they control, so you don't get to touch it without "jailbreaking" it. Jailbreaking is done by exploiting a security flaw, and it's a cat and mouse game with Apple fixing bugs to close holes and hackers finding new ones to regain control of the devices. Not a game I want to play just to have control over my own hardware. This 100% rules out iOS for me, it's a complete dealbreaker. No matter how shiny, how bug free, how capable it is I wouldn't trade unless there was absolutely no alternative.</p>
Android's secure boot chain</h3>
Android also secure their boot chain by default, however whether you can unlock it and flash a different recovery/bootloader/OS/ROM depends on the phone manufacturer, individual device, and sometimes the phone network provider (aka locking/unlocking).</p>
This means choose your device and provider carefully if you wish to unlock without wasting your time trying to jailbreaking a locked device.</p>
Artificial app limitations when unlocked</h3>
Android has a system that allows apps to hide or limit functionality called SafetyNet</a>. For apps like NetFlix that won't even show in the play store for unlocked phones we're into the mega-discussion that is DRM</a>. In short if a content creator wants to lock their device to a locked-down platform that's fine be me but I won't be using those platforms for my main computing, instead I might have them as throw-away additional appliances like the google chromecast, or samsung TV's built in junk. What really matters here is what alternatives you have available to you. For google pay, I'm very slightly sad to miss out but my contactless debit card still works fine so I'll live.</p>
Hardware choices</h2>
Given the "app" problem, any devices not capable of running Android or a derivative are currently out of the running (sad times).</p>
A non-negotiable for me is manufacturer support for replacing the bootloader and everything above that (OS, apps). This ensures the life of the device beyond the point at which the manufacturer loses interest or does something horrific/stupid. This is also a signal for who is truly in charge of the device once it's "yours" - you the user or the supplier and their developers. By locking you out of making modifications it's a clear signal that you are not to be trusted with the power over your own device.</p>
As the useful innovation in hardware slows down (2G to 3G was a big jump, but 8-megapixels camera to 32-megapixel cameras is less foundational) it will be less often that devices will become obsolete through huge leaps in capability. It will also mean that phone hardware will inevitably become more commodity and less proprietary, giving a chance for more open standards to get a foothold (just as x86 replace proprietary CPU architectures many decades ago), while still being acceptably capable. We shouldn't be throwing out devices because the manufacturer decided not to provide security patches for their OS variant, and certainly not because the manufacturer took away a feature we like through forced updates.</p>
Google Nexus / Pixel (meh)</h3>
I have run a google nexus</a> 4 previously which was flashable and rootable, but long since obsolete in hardware capabilities (and suffered a hardware failure). I never looked back at the google line after I discovered OnePlus, I can't comment on whether the current pixel phones are a good device for my constraints. I don't particularly want to further feed the beast. If you want the authentic pure google experience I guess these are probably worth a look but the pure google experience seems less and less appealing as the years roll on.</p>
Sony (yuck)</h3>
Someone told me that a particular Sony phone was flashable, I researched it, found instructions on flashing it, ordered one... and couldn't flash it. Discussing with Sony support it transpired that that precise model was sometimes flashable and sometimes not, and the only way to find out was to order one and look at the serial number. Fuck you Sony. They never have understood the open source movement and look like they still don't. They always treated their kit like appliances not a blend of hardware and software (or firmware) with everything that means for the long term management of the software. It's a shame because they do make nice hardware. I'd still buy a dumb Sony audio amplifier but not much else of theirs. Returned and refunded, try again...</p>
OnePlus (yay, no wait, boo)</h3>
This is my choice for the new OnePlus 9 Pro</a> I'm trying to set up, and the OnePlus 5T</a> I've been using for more than a couple of years now.</p>
They don't make the grade for the pure "everything must be open including chip bytecode and chip architecture" as they use proprietary chips, but they make the right balance for me of reasonably high spec hardware for a reasonable price, android compatibility, and the ability to flash whatever ROM/OS you like (at your own risk).</p>
I've been running a OnePlus 5t for a while now (bought outright, no network lockin or contract nonsense) and generally happy enough with it. There are some bugs and storage is starting to constrain hence the upgrade, plus the camera could be better by modern standards and I don't fancy carrying a separate camera.</p>
I now have had the OnePlus 9 Pro sat in a drawer for two months waiting for me to figure out what-the-FSM</a> to run on it, hence this blog post. It's already been unlocked and I've tried flashing a couple of things but am not happy with the setup yet. I stopped looking at it while I concentrated all my efforts on making sure my business was going in the right direction (or any direction at all).</p>

To get root you have to wipe the phone</p>
</blockquote>
The OnePlus comes with OxygenOS (an android derivative) preinstalled, but not rooted and with a locked bootloader. To get root you have to wipe the phone which means it's important to do it before you start using it. It's caused me no end of pain that you can't use titanium backup</a> to backup all your data without root, and you can't get root without losing your files. I understand why (so you can't just unlock a phone to steal all the data) but that doesn't make it less of a pain in the arse. Bonus points for operating system updates (over the air / OTA or just normal updates) de-root the phone.</p>
XDA Developers detailed page on the OnePlus 9 including pro</a></p>

But wait, there's trouble in paradise.</p>
The founder Carl Pei left OnePlus in 2020</a> after 7 years at the company.</p>
OnePlus was funded by an existing phone company called Oppo</a>. (And both OnePlus and Oppo are part of BBK</a>)</p>
It seems that OnePlus is basically now dead in the water</a> as a brand if not a company in its own right and will be sucked back into whatever the mothership has to offer, breaking everything along the way by trying to merge Oppo's colorOS with OnePlus's OxygenOS into an unholy mess.</p>
Fuck.</p>

As if that wasn't enough:</p>

"OnePlus is adding artificial limitations and breaking features via software updates, and there are no indications that they'll improve." ~ GCam Hub, https://www.celsoazevedo.com/files/android/google-camera/f/post-05/</a></p>
</blockquote>

It turns out that the founder of OnePlus, Carl Pei, has set up a new company called "Nothing</a>" (enjoy googling that one), which did just make headphones (unexciting), but who have just announced a phone. So maybe this is the true successor to the likely dead OnePlus brand. https://www.theverge.com/2022/3/23/22992424/nothing-phone-1-smartphone-carl-pei-apple-ecosystem</a></p>
So what OS options does that leave?</h2>
So not iOS and preferably not Android either.</p>
Unfortunately the apps mean that I'm stuck with Android and its derivatives for the time being.</p>
It seems the best hope for salvation is the Linux based options discussed above eventually getting sufficient app coverage to make the jump more bearable, or maybe even some kind of android emulation allowing them to run android apps sufficiently well. The path from Windows to Linux for me included a mix of dual-boot, virtualization and wine (an api compatibility layer, not an emulator)</p>
It's worth mentioning that some (most/all?) phone manufacturers make proprietary customizations to Android or even complete forks</a> that you may or may not want. Samsung for example (for the Galaxy line of phones) replaces the "launcher" with the "Samsung Experience"</a>. I'm going for more open rather than less, so I'm not interested in trading slickness for losing even more control. I just want a platform that runs the apps I care about and is as open as possible.</p>
Researching the options</h3>
I don't follow this stuff all the time so I had to do some googling (duckduckgo</a>-ing). Here's some useful comparisons I turned up:</p>

It's FOSS, Open Source Mobile OS Alternatives To Android</a> - mostly pure Linux things that don't have the android app ecosystem</li>
PC Magazine, Break Away From Android: 7 Free Open-Source Mobile OSes to Try</a></li>
GetDroidTips has some more ROMs I haven't looked at</a></li>
</ul>
I've got more to learn/research here so I'll expand this section as and when I learn more. Think of this as a bliki</a>. There's a full history in git (link at end).</p>
Towering stacks of patches</h3>
The below are probably the most realistic alternatives to stock Android and iOS for those who want more openness but don't want to give up all the modern conveniences in the name of openness and/or privacy. They are however all customizations of Android and that brings a very real problem.</p>
The core open "Android Open Source Project (AOSP)" is run by google for google's benefit and for their own Android ecosystem. There is significant engineering effort continually poured in to this, with major releases made on remarkably short timelines.</p>
Any project that makes a customization to AOSP such as LineageOS and keeps track of it in their own fork is constantly at the mercy of changes to the foundation it has been based on, a foundation which really has no reason to care about them.</p>
This creates a never ending challenge that stunts innovation outside the Android/iOS duopoly. If you build a significant customization, it will sooner or later be broken and need re-engineering because of changes in AOSP, or maybe even be impossible to resurrect. The more you innovate and customize, the more engineering fire-power is required just to stand still on the treadmill of change in the platform.</p>
While we have good tools for managing all this complexity (git, gerrit) this is a fundamental and unavoidable problem with building on a platform that doesn't care about you.</p>
Contrast that with the Linux ecosystem where the platforms people are building on (consisting of many many layers and teams) are much more interested in supporting downstream projects and allowing choice. Projects that run on Linux can often run with minimal modification for decades. Linux is also not built for the benefit of some particular vendor like google with their own agenda, it is instead run by a foundation with far more broad interests and pressures.</p>
Android derivative options</h3>
I now need to look into the below and decide what to run (maybe with some trial flashing). Once that's done it'll be on to the slog of setting up the actual device how I want it.</p>
OxygenOS from OnePlus - used to be good, now dying a painful death</h4>
See the section on OnePlus hardware above. The company and OS are in trouble now the founder has left and the operating system has been merged with Oppo's colorOS (read "fucked")</p>

https://www.oneplus.com/uk/oxygenos</a></li>
</ul>
This is the default for the OnePlus device I have, which means it's the most likely to make the hardware perform at its best.</p>
After a wobble on data reporting</a> (which I'm not too bothered about) they seem to have largely sorted out their privacy game... apart from being full of google services just like the rest of the mainstream android devices. To play or not to play...</p>
Here's the original article disclosing OnePlus silently sending data home</a>.</p>
It seems the latest news is that Oxygen is falling apart. XDA Developers: "OxygenOS 12 for the OnePlus 9 series is littered with bugs"</a></p>

"It was shit after merge with #Oppo and getting worst day by day. I really like #OnePlus device but #OxygenOS is dead now." ~ @rahulawanjari, 9 Dec 2021, twitter</a></p>
</blockquote>
Crap, there goes another good option. As if the mobile phone space wasn't horrific enough already.</p>
In fact OxygenOS 12 was so bad, that they've just OnePlus pulled the plug on the entire OxygenOS 12 upgrade!!</a></p>
Bonus points for the anti-competitive practice of actively denying access to hardware features (secondary cameras)</a>. I hope the courts slap them down for this clearly anti-competitive behaviour.</p>

So after discovering that shit-show-in-paradise from OnePlus I could potentially skip OxygenOS 12 and stick to the less f*cked OxygenOS 11, but that doesn't seem like much of a plan so it looks like I'll be skipping any fancy hardware capability and looking for a more open and less broken Android variant. Yet more proof that corporations can't be trusted as guardians of our software for the long term without GPL or at least MIT to stop them from hurting their users sooner or later. Did I mention my podcast is called "Software Should Be Free". Does this horror show give you a hint as to why it's called that?</p>
LineageOS (yep)</h4>
LineageOS</a> is the successor to the now defunct CyanogenMod. See the history of LineageOS and it's relationship to CyanogenMod here on Wikipedia</a>.</p>
CyanogenMod was a bastian of freedom and innovation. I hope Lineage has managed to continue that.</p>
There does appear to be a build of LineageOS for the OnePlus 9 pro</a> - I think this is worth trying out.</p>
LineageOS has it's own recovery image, so no need to run TWRP as well.</p>
Review of LineageOS on XDA Developers</a></p>
Lineage for microG (maybe?)</h4>

https://lineage.microg.org/</a></li>
https://github.com/lineageos4microg</a></li>
</ul>
This is a fork of lineage with microG already set up properly and a patch that lineage wouldn't allow that let's them spoof google signatures in order to trick apps into believing they are using the real google play services</p>

Q: "Why do we need a custom build of LineageOS to have microG? Can't I install microG on the official LineageOS?"</p>
A: "MicroG requires a patch called "signature spoofing", which allows the microG's apps to spoof themselves as Google Apps. LineageOS' developers refused (multiple times) to include the patch, forcing us to fork their project."

~ Lineage for microG faq</a></p>
</blockquote>
Replicant (not yet)</h4>
"Replicant</a> is a fully free Android distribution running on several devices"
I'm glad this freedom is here but I'm not quite ready to give up every last proprietary thing until there's a bit less of a gap.</p>
Supported devices</a> shows purely Samsung Galaxy devices currently.</p>
/e/ (aka Eelo)</h4>
https://e.foundation/</a></p>

"We build desirable, open source, privacy-enabled smartphone operating systems."</p>
</blockquote>
This Android derivative is focussed on privacy; eliminating as much data collection (i.e. all things google) as possible. Laudable but not my main focus right now.</p>
You can buy their reconditioned phones with their OS pre-installed or just install the OS on something else.</p>
They provide their own alternative cloud hosting for contacts etc for reasonable prices.</p>
Their dev build has a root option it seems which might be useful. - https://doc.e.foundation/support-topics/root-e-os.html</a></p>

/e/ on wikipedia</a></li>
itsfoss.com/eelo-mobile-os</a></li>
https://www.pcmag.com/news/google-free-android-smartphones-are-now-available-in-the-us</a></li>
</ul>
GrapheneOS (nope)</h4>

"GrapheneOS</a> - The private and secure mobile operating system with Android app compatibility. Developed as a non-profit open source project."</p>
</blockquote>
GrapheneOS only officially supports Pixel phones</a></p>
CalyxOS (nope)</h4>

CalyxOX</a> - "Your Phone Should Be Private Everyone needs a phone. Not everyone wants to be spied on. Reclaim your privacy with CalyxOS."</p>
</blockquote>
Supports Pixel devices and one Xiaomi phone (never heard of it).</p>
Seems to be pushing Signal and Tor so is super-privacy focussed.</p>
Google Apps and Services</h2>
There are two major parts to this:</p>

Google play - the app store for downloading apk files to install (plus ratings, reviews, screenshots etc)</li>
Google services - these provide services that many apps want to use to avoid multiple implementations such as location information, push notification etc.</li>
</ol>
There's a few ways of getting the google play store and related proprietary horrors:</p>

Open GApps - pulls down the proprietary apps and writes them to the image</li>
MindTheGapps? - pulls down the proprietary apps and writes them to the image</li>
microG - open source re-implementation of the services</li>
</ul>
For Lineage 18.1 (Android 11) the Lineage wiki links to MindTheGaps</a>, and XDA Developers says "Always Use the GApps Package Recommended by your ROM Developer"</a></p>
I don't mind proprietary software as long as there's choice out there, and the apple app store + google play duopoly on apps is not choice. F-Droid is fine but doesn't have a single big name vendor's apps, so it's probably enough for google to dodge an anti-competitive lawsuit but not enough to produce any real competition.</p>
Comparisons:</p>

https://www.reddit.com/r/LineageOS/comments/8358p0/mindthegapps_vs_opengapps/</a></li>
https://www.reddit.com/r/degoogle/comments/olsv4c/i_am_very_confused_by_opengapps_mindthegapps/</a></li>
</ul>
Open GApps (installer)</h3>
Installer for google's proprietary services.</p>

GApps https://wiki.lineageos.org/gapps</a></li>
https://github.com/opengapps/opengapps</a></li>
https://opengapps.org/</a></li>
</ul>
Q: Why is this such a pain? A: Licensing...</p>

"Due to licensing restrictions, these apps do not come pre-installed with ROMs others than those from vendors that are part of the Open Handset Alliance</a> and must be installed as a sideload package by the user themselves." ~ https://github.com/opengapps/opengapps/wiki/FAQ</a></p>
</blockquote>
We're lucky they let us do this at all, google could take their toys away from us at any time.</p>
For Open GApps pico looks sufficient https://github.com/opengapps/opengapps/wiki/Package-Comparison</a>, Lineage recommends nano and no bigger.</p>
MindTheGapps (installer)</h3>
Installer for google's proprietary services.</p>
Really haven't found much information about this.</p>
About</h4>

https://gitlab.com/MindTheGapps/vendor_gapps</a> - I think this is probably the official source

There's no download link, as mentioned in this issue: https://gitlab.com/MindTheGapps/vendor_gapps/-/issues/1</a></li>
https://github.com/MindTheGapps/vendor_gapps</a> appears to be a stale copy, maybe they moved hosting to gitlab</li>
</ul>
</li>
https://www.getdroidtips.com/mindthegapps-8-1-0/</a></li>
https://forum.xda-developers.com/t/change-opengapps-for-mindthegapps-afterthought.3837816/</a></li>
</ul>
Installation</h4>

This wasn't a lot of help but basically said just flash it: https://www.getdroidtips.com/mindthegapps-8-1-0/</a></li>
</ul>
microG (open source reimplementation)</h3>
https://github.com/microg</a></p>
A free software implementation of shared services provided by the proprietary google services (such as location and push messaging used by many apps).</p>
Doesn't work properly when flashed to lineage according to the faq so they made their own fork of Lineage "Lineage for microG" (see above)</p>
Aurora - play store proxy</h3>
Download apks (installation files) from the google play store without any googleness.</p>

https://aurora-store.en.uptodown.com/android</a></li>
https://f-droid.org/packages/com.aurora.store/</a></li>
</ul>
Yalp - play store proxy</h3>
https://yalp-store.en.uptodown.com/android</a></p>
F-Droid app store</h3>
F-Droid</a> is an app store for android like google play but without all the googlyness.</p>
This a bit pointless if you install play because all the open source apps are available in both, and the closed ones are only in play. I think I'll skip this for now but I'm glad it exists... Though it is pre-installed in lineage-for-microG so I have poked around and installed a few things.</p>
Playmaker - play store / f-droid integration</h3>
https://github.com/NoMore201/playmaker</a></p>
Recovery images</h2>
TWRP (great but not needed)</h3>
Previously I've used TWRP, and I got as far as flashing it before discovering Lineage has it's own recovery image</a></p>

Unlock bootloader https://www.getdroidtips.com/oneplus-9-pro-unlock-bootloader/</a></li>
Download TWRP for lemonadep https://twrp.me/oneplus/oneplus9pro.html</a> - 3.6 twrp download for lemonadep</a></li>
Get twrp public gpg/pgp public key to be able to verify .asc</code> files https://twrp.me/faq/pgpkeys.html</a></li>
Flash TWRP (TeamWin Recovery Project?) https://twrp.me/</a>

Power off the phone</li>
Hold down volume-down and power buttons</li>
Select english</li>
select reboot to fastboot</li>
connect the usb cable (usb-c to usb-c doesn't work on my dell xps, use the bigger usb 3 A to usb-c cable)</li>
run fastboot devices</code> on the laptop (connected over usb), you should see your phone listed</li>
boot the phone from the local twrp image fastboot boot twrp-3.6.0_11-0-lemonadep.img</code>, phone boots up into twrp gui</li>
advanced > flash current twrp</li>
there's a note about bootloops, but the options isn't there so skipping that</li>
reboot > power off (says current slot: A, just for the record)</li>
power on with volume-down held again, boots straight to twrp this time</li>
</ul>
</li>
</ul>
LineageOS's recovery image</h3>
Came across this here: https://www.xda-developers.com/lineageos-18-1-review/</a></p>
Works well, flashed no problem and have used both recovery and fastboot modes with no issues. You'll see it as part of the steps below.</p>
Installing LineageOS on OnePlus 9 Pro (take 1)</h2>
It begins. LineageOS</a>.</p>
The oneplus 9 pro device is codename lemonadep</code>.</p>
Useful howtos:</p>

LineageOS official installation wiki page for OnePlus 9 pro lemonadep</a></li>
How-To Geek: How to Install LineageOS on Android</a></li>
Android Authority: Beginner’s guide to installing LineageOS on your Android device</a> - lots more context and screenshots</li>
LineageOSROMS (unofficial)</a> - a very short 8 step guide

LineageOSROMS (unofficial) full guide</a></li>
</ul>
</li>
GetDroidTips</a> - not sure there's much extra useful here</li>
</ul>
XDADevelopers coverage of Lineage support for OnePlus 9 pro</a></p>
The steps:</p>

Download latest nightly (there's no stable/unstable on this): https://download.lineageos.org/lemonadep</a> - this has both the ROM (OS image) and the recovery image.</li>
Download copy-partitions https://androidfilehost.com/?fid=2188818919693768129</a> (as per wiki)

The sha256 for the copy I have is 200877dfd0869a0e628955b807705765a91e34dff3bfeca9f828e916346aa85f  copy-partitions-20210323_1922.zip</code></li>
</ul>
</li>
Verify all the sha256 sums: sha256sum -c *.sha256</code></li>
Boot to fastboot</li>
flash the lineage recovery:tim@max:~/Downloads/oneplus9pro/LineageOS</span></span>
$ fastboot flash boot lineage-18.1-20211214-recovery-lemonadep.img</span></span>
Sending 'boot_a' (196608 KB)                       OKAY [  6.113s]</span></span>
Writing 'boot_a'                                   OKAY [  0.655s]</span></span>
Finished. Total time: 6.985s</span></span></code></pre></li>
copy partitions as instructed

boot into recovery (vol-down + power)</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload copy-partitions-20210323_1922.zip</code></li>
Ignore unknown sig and "continue"</li>
</ul>
</li>
wipe

boot into recovery (vol-down + power) if not already in it</li>
"format data / factory reset"</li>
</ul>
</li>
OS install

boot into recovery (vol-down + power) if not already in it</li>
"apply update" > "apply from adb" (aka sideload)tim@max:~/Downloads/oneplus9pro/LineageOS</span></span>
$ adb sideload lineage-18.1-20211214-nightly-lemonadep-signed.zip</span></span>
serving: 'lineage-18.1-20211214-nightly-lemonadep-signed.zip'  (~47%)</span></span>
	adb: failed to read command: Success</span></span></code></pre>apparently this is normal, but the failed to flash error on the phone is not, and rebooting drops me back into the existing Oxygen install. Dammit.... Try again</li>
Second attempt ran fine

output: step 1/2</code></li>
output: step 2/2</code></li>
then recovery logo comes back up and it just sits there</li>
volume up to the back arrow at the top, power to press it</li>
then</em> it says "install complete with status 0</code>" (zero being unix-speak for no-issues)</li>
but it doesn't ever get past the lineage logo when booting, maybe because of the boot of oxygen. hmm</li>
hard-power-off</a> by holding down volume-up and power button</li>
</ul>
</li>
take 3,

back to recovery</li>
re-wipe (factory reset)</li>
reboot</li>
this time it booted... briefly</li>
it got to setup screen, then before I did anything rebooted into some kind of recovery failsafe and prompted to factory reset again, which I did. This time it rebooted and stayed up while I zipped through the wizard.</li>
</ul>
</li>
</ul>
</li>
go take a look - ignore the warning about doing gapps etc first just so we can see what vanilla Lineage looks like. We can always wipe & reflash

wonderfully empty app list!</li>
basic main camera & selfie cam works</li>
flashlight works</li>
odd android verify notification, didn't seem to work</li>
top shelf only available on lockscreen, odd</li>
no sim yet and wifi not connected yet, so not much else to test</li>
vol-down + power still takes screenshots</li>
</ul>
</li>
</ul>
Connect adb, see if we get root shell</h3>
https://developer.android.com/studio/command-line/adb#Enabling</a></p>

On the phone enable adb

settings > about phone > tap "build number" repeatedly</li>
settings > system > advanced (grrrr) > developer options

usb debugging [on]</li>
rooted debugging [on]</li>
disable adb authorization timeout [on]</li>
</ul>
</li>
connect usb cable to laptop</li>
phone prompts "Allow USB Debugging?"

tick "always allow"</li>
press "allow"</li>
</ul>
</li>
</ul>
</li>
On the laptop adb devices</code> should now show the phone</li>
</ul>
Success with non-root:</p>
$ adb shell</span></span>
OnePlus9Pro:/ $ ^D</span></span></code></pre>
Root denied without flipping the "rooted" option:</p>
$ adb root </span></span>
ADB Root access is disabled by system setting - enable in Settings -> System -> Developer options</span></span></code></pre>
After enabling rooted debugging as above:</p>
$ adb root  </span></span>
restarting adbd as root</span></span>
$ adb shell</span></span>
OnePlus9Pro:/ # whoami</span></span>
root</span></span></code></pre>Getting root back</h3>
Grumble</h4>
It really annoys me that on a laptop this isn't even a step, yet on Android this is a whole additional drama. But I may have already mentioned that. Understanding the reasons doesn't mean I have to like it.</p>
adb root shell</h4>
According to https://www.xda-developers.com/lineageos-dropping-superuser-addonsu-implementation-favor-magisk-manager/</a> adb root shell is the official LineageOS way of getting root access to "mess with important files".</p>
$ adb root  </span></span>
restarting adbd as root</span></span>
$ adb shell</span></span>
OnePlus9Pro:/ # whoami</span></span>
root</span></span></code></pre>
Worth knowing but not much help for running titanium backup as root.</p>
Magisk</h4>
According to https://www.xda-developers.com/lineageos-dropping-superuser-addonsu-implementation-favor-magisk-manager/</a> Magisk is the only real option now. I guess SuperSU still works but lacks the masking that allows you to fool android pay etc.</p>
⚠️ Beware of fake Magisk sites !! ⚠️</h5>
Github is the official home for Magisk and it's downloads: https://github.com/topjohnwu/Magisk</a></p>
Beware of clone sites</strong></a> especially ones offering downloads.  MagiskManager.com</code> is not legit or authorised and may host malware at some point. I've seen a few others too. They look like SEO optimised clones that could potentially be feeding some or all visitors malware.</p>
It's worth being extra-careful with things like recovery images and rooters because by definition they have full control of your device, malicious or malign.</p>
Magisk root install</h5>
LineageOS said not to boot before doing installing GApps, I'm not sure if that applies to Magisk too. I already have booted up. Not sure if a factory reset plus wipe will be enough or whether the image will have to be reflashed. I'll try with just an install, if that doesn't work I'll try the reset and see what happens, and if that</em> doesn't work I'll try a full reinstall.</p>

Look at readme at https://github.com/topjohnwu/Magisk</a></li>
Follow instructions https://topjohnwu.github.io/Magisk/install.html</a></li>
Download latest release https://github.com/topjohnwu/Magisk/releases/latest</a></li>
install the app with adb (with the phone on, and booted to LineageOS as per normal use)

https://www.droidviews.com/install-apk-files-using-adb-commands/</a></li>
</ul>
$ adb install Magisk-v23.0.apk </span></span>
Performing Streamed Install</span></span>
Success</span></span></code></pre>

On the phone, swipe up for the full app list (drawer), and there is a Magisk icon!</li>
</ul>
</li>
</ul>
Getting boot.img</h6>
The instructions for Magisk just say you need boot.img</code>, leaving know clues how to get it. I've proposed explaining how to extract them in this pull request</a></p>
It turns out it's embedded in the Lineage system image lineage-18.1-20211214-nightly-lemonadep-signed.zip</code> that we've already downloaded.</p>
There's another payload extractor with easier dependencies that's worth a try, especially if you don't already have python: payload-dumper-go</a>
| payload-dumper-go downloads</a>. I've now tested this and it works great. You don't even have to unzip the zip.</p>
Extract the payload.bin</code> file from the lineage-18.1-20211214-nightly-lemonadep-signed.zip</code> to the same folder as the python script.</p>
Some instructions on using python to extract the boot image: https://forum.xda-developers.com/t/guide-how-to-extract-payload-bin-from-ota.3830962/</a> (ignore the file it's older than the github one)</p>
The payload_dumper</code> file appears to be from https://github.com/vm03/payload_dumper</a> assuming my googling is accurate.</p>
I didn't need to do the pip install.</p>
I did change the python to python3 at the top of the py file and make it executable.</p>
tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span>
$ ./payload_dumper.py payload.bin </span></span>
boot</span></span>
dtbo</span></span>
odm</span></span>
product</span></span>
system</span></span>
system_ext</span></span>
vbmeta</span></span>
vbmeta_system</span></span>
vendor</span></span>
vendor_boot</span></span>
tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span>
$ ll output </span></span>
total 5.6G</span></span>
-rw-rw-r-- 1 tim tim 192M Dec 20 23:20 boot.img</span></span>
-rw-rw-r-- 1 tim tim  24M Dec 20 23:20 dtbo.img</span></span>
-rw-rw-r-- 1 tim tim 2.8M Dec 20 23:20 odm.img</span></span>
-rw-rw-r-- 1 tim tim 1.3G Dec 20 23:21 product.img</span></span>
-rw-rw-r-- 1 tim tim 267M Dec 20 23:21 system_ext.img</span></span>
-rw-rw-r-- 1 tim tim 2.5G Dec 20 23:21 system.img</span></span>
-rw-rw-r-- 1 tim tim 8.0K Dec 20 23:21 vbmeta.img</span></span>
-rw-rw-r-- 1 tim tim 4.0K Dec 20 23:21 vbmeta_system.img</span></span>
-rw-rw-r-- 1 tim tim 192M Dec 20 23:22 vendor_boot.img</span></span>
-rw-rw-r-- 1 tim tim 1.3G Dec 20 23:22 vendor.img</span></span></code></pre>
Hurrah, now we have a boot.img</code> to give to Magisk.</p>
Magisk rooting</h6>
Copy the boot.img</code> across:</p>
tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span>
$ adb push output/boot.img /sdcard/ </span></span>
output/boot.img: 1 file pushed, 0 skipped. 177.5 MB/s (201326592 bytes in 1.082s)</span></span></code></pre>

Press install on the Magisk app.</li>
"Select and Patch a File"</li>
hamburger in file browser > "OnePlus 9 Pro" (obviously, duh) > boot.img</code></li>
"let's go ->"</li>
Magisk shows steps then" output file is written to..." and a path to download folder</li>
</ul>
tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span>
$ adb shell</span></span>
OnePlus9Pro:/ # ls -lh /sdcard/Download/</span></span>
total 96M</span></span>
-rwx------ 1 u0_a163 u0_a163 192M 2021-12-14 13:21 magisk_patched-23000_BYMDt.img</span></span></code></pre>

pull the file back to the laptop adb pull /sdcard/Download/magisk_patched-23000_BYMDt.img</code></li>
Now flash the file to the phone:

(the vol-up + power didn't work for me, just did a normal boot, vol-down + power is reccovery not fastboot)</li>
Settings > system > advanced (ffs) > gestures (wtf?) > power menu > advanced restart [on]</li>
power button (hold) > power > restart > fastboot</li>
</ul>
</li>
</ul>
tim@max:~/Downloads/oneplus9pro/payload_dumper</span></span>
$ fastboot flash boot magisk_patched-23000_BYMDt.img </span></span>
Sending 'boot_b' (196608 KB)                       OKAY [  6.035s]</span></span>
Writing 'boot_b'                                   OKAY [  0.639s]</span></span>
Finished. Total time: 6.853s</span></span></code></pre>

power off</li>
power on</li>
open up the Magisk app - it shows installed version number now, hurrah</li>
</ul>
Using Magisk:</p>

Click the cog (top right) to optionally turn on "MagiskHide"</li>
Along the bottom are four icons

Home</li>
Shield - shows root requests</li>
bug - shows logs</li>
jigsaw piece - shows modules, activate, deactivate & install</li>
</ul>
</li>
</ul>
adb</h3>
adb</a></p>
I consider this an important backup access to the phone. If the screen fails then this allows you to pull any important files off over usb.</p>
Steps for this are above.</p>
Installing "Lineage for microG" (tried, rejected for now)</h2>

https://lineage.microg.org/</a> has install and download links</li>
Download image from https://download.lineage.microg.org/lemonadep/</a>

check sha: sha256sum -c lineage-18.1-20211220-microG-lemonadep.zip.sha256sum</code></li>
run update_verifier python script:

download (or clone) python update_verifier</a> - https://github.com/lineageos4microg/update_verifier/archive/refs/heads/master.zip</a></li>
setup python3 with asdf-vm</a> (I had python 3 but no pip, and I like to use adsf)</li>
extract the zip, open terminal in extracted folder</li>
asdf plugin add python</code></li>
asdf install python 3.10.1</code></li>
asdf local python 3.10.1</code></li>
pip install -r requirements.txt</code></li>
python update_verifier.py lineageos4microg_pubkey ../lineage-18.1-20211220-microG-lemonadep.zip</code></li>
output: verified successfully</code></li>
</ul>
</li>
</ul>
</li>
Go back to lineage install instructions https://wiki.lineageos.org/devices/lemonadep/install</a></li>
Boot to fastboot</li>
flash the lineage-microG recovery: fastboot flash boot lineage-18.1-20211220-microG-lemonadep-recovery.img</code></li>
copy partitions as instructed

boot into recovery (vol-down + power)

or "enter recovery" from lineage fastboot</li>
</ul>
</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload ../LineageOS/copy-partitions-20210323_1922.zip</code></li>
Ignore unknown sig and "continue"</li>
</ul>
</li>
wipe

boot into recovery (vol-down + power) if not already in it</li>
"format data / factory reset"</li>
</ul>
</li>
OS install

boot into recovery (vol-down + power) if not already in it</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload lineage-18.1-20211220-microG-lemonadep.zip</code></li>
</ul>
</li>
reboot</li>
skip all the guided setup steps</li>
open the microG app</li>
run the self check (all green ticks for me, woo)</li>
</ul>
So now I have a clean LineageOS install with a FOSS re-implementation of google's proprietary shared services. Win.</p>
Now re-run magisk sideload from above to get root again.</p>

re-enable adb as before</li>
re-enable advance power button menu</li>
adb install Magisk-v23.0.apk</code></li>
grab a payload-dumper-go binary release</a></li>
payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go lineage-18.1-20211220-microG-lemonadep.zip</code></li>
adb push extracted_20211221_231517/boot.img /sdcard/Download/</code></li>
run the patch in the phone Magisk UI (under "install")</li>
get the filename to pull: adb shell ls /sdcard/Download</code></li>
adb pull /sdcard/Download/magisk_patched-23000_d5mu3.img</code></li>
reboot to fastboot</li>
fastboot flash boot magisk_patched-23000_d5mu3.img</code></li>
"reboot system now" in lineage fastboot menu</li>
</ul>
Success, now I have root and</em> microG.</p>
Next, see if Aurora does the job, and if not how do I get google play on top of microG, if that's even possible. Till next time...</p>
Getting apps</h3>
F-Droid</h4>
F-Droid is preinstalled (stared blankly at app list for a while before spotting it!) so that was easy, this can install many</em> open source apps, and warns about "anti-features" which is a nice touch.</p>
Used F-Droid to install andOTP as a test (alternative to google authenticator 2FA app). Worked well.</p>
Apps from play store - Aurora</h4>
Found Aurora in f-droid with a search, installed no problems, opened it up.</p>
First thing it does is prompt for install type which is confusing and lacks enough context to know what it means. The four types are explained further here: https://auroraoss.com/faq/#how-does-aurora-store-install-apps</a> thought I'm still not really sure. I don't know what a "split apk" is yet, and I don't see if there's a difference between root and services as I could do either and they both say they do the same thing.</p>

Session - recommended but not clear why, I think it's saying it's like "Aurora Services" but with more something-or-other</li>
Native - doesn't support split apks, that sounds bad, see below for what they are (see below for split apk research)</li>
Root - auto install when downloaded</li>
Aurora Services - auto install when downloaded</li>
</ul>
How to choose?  I guess I'll go with the recommendation.</p>
Couple of theme questions. Meh.</p>
Installer question: "Select a suitable installer"</p>

External Storage Access [grant] - pops and allow dialog, clicked "allow"</li>
External Storage Manager [grant] - required on R+ (aka android 11), which we are running, guess I'll need that, - opened permissions screen, toggled to allow</li>
Installer Permission [grant] - opened permissions screen, toggled to allow</li>
</ul>
"Finish"</p>
Log in using:</p>

Google</li>
Anonymous</li>
</ul>
It seems from what I've read in the FAQ and terms that using the real login is best because the anon accounts end up with random locales and app restrictions depending on where the login was generated.</p>
[install keepassdroid and push my kdb across with adb to get the google login, eventually the kdb will be sync'd with syncthing]</p>
... log in with google ...</p>
And we're in, app listing showing. Disconcerting to see disney+ etc in "free" app store on a "free" phone.</p>
WhatsApp isn't available in f-droid so let's try that as an experiment... It's in the list... it installed... it opened... it worked! (And logged me out of the other phone, fucking whatsapp).</p>
What's a split APK?</h5>

APK splitting - https://developer.android.com/studio/build/configure-apk-splits</a></li>
Android App Bundles - defers apk generatoin to play store https://developer.android.com/guide/app-bundle</a></li>
Feature delivery - partial app additions and updates https://developer.android.com/guide/playcore/feature-delivery</a></li>
Old unsupported app "SAI" split apk installer:

https://forum.xda-developers.com/t/guide-split-apk-installation.3934631/</a></li>
https://github.com/Aefyr/SAI</a></li>
</ul>
</li>
https://www.reddit.com/r/androiddev/comments/cenuvs/what_is_it_with_publishing_split_apks_on_google/</a></li>
https://www.reddit.com/r/androiddev/comments/ci4a7r/confused_about_split_apk/</a></li>
</ul>
The state of the system</h3>
What's working and what isn't.</p>
Google voice to text (dead)</h4>
The keyboard has a little mic button which allows you to use google's services to enter text anywhere you like by typing. This is useful as I hate phone keyboards. I use it quite a lot. Unsurprisingly with the microG setup that's not working at all. The button is there but it does nothing.</p>
Bluetooth (working)</h4>
Headphones connected and played no problem.</p>
The state of the apps</h3>
What's working and what isn't.</p>
Banking apps (mixed)</h4>
Let's start with the big one. This is most likely to sink an open source effort as they try to protect themselves by joining in the lockdown.</p>
Surprisingly two out of three of the banking apps I'd like to use actually worked on the rooted custom ROM with an unlocked bootloader. I had not turned on the magisk hide capability at this point. One banking app behaved as if nothing was different, one popped a warning but allowed the app to continue and one flat out refused to run. I'm not going to name the banks involved on the public internet as I don't want to encourage anyone to test the security of my banking.</p>
One of the apps logged me out of my other phone. The other functioning app offered to transfer a "digital key" to the new device, and without that allowed me to use the first phone to generate a login code and use that to poke around. I didn't try transferring the key.</p>
More worrying was the realisation that I'd been locked in to running a google/iOS locked-down device by one of the banks as they are an "app-first" bank. You can't even use the online banking without the phone app, and the phone app requires a locked and signed phone. It's only by going on this very journey of actually trying to set up a phone that I control that I'd even noticed this insidious creeping embrace of platform-lockin. So now to truly escape the platform, I have to change banks as well.</p>
Google calendar (mostly working)</h4>
I'm afraid I still use google's calendar, and there's a high cost of change as I've got my other half using a shared calendar. The calendar worked fine, although oddly the "widget" (view of calendar that can be placed on the phone's desktop when unlocked) was blank until I left the phone in the drawer with wifi off for two days. For who-knows-what reason when I got it out the drawer it has started working.</p>
Oddly the events have shown up but reminders are missing. This is a bit of a problem as I use them a lot. It's not that they're local to my other phone because they show on the google calendar web interface too.</p>
Amazon things (working)</h4>
The amazon shopping app worked no problem.</p>
To my surprise the amazon primve video app worked and played content as normal. I rather expected some kind of DRM smackdown to kick in on a rooted device.</p>
Audible worked and played no problem.</p>
Spotify (working)</h4>
Again pleasingly working with no quibbling or warnings. This is more important to me at the moment than perhaps it should be.</p>
Casting sound to chromecast audio not working.</p>
Whatscrapp (working)</h4>
Much as I hate WhatsApp and it being owned by Facebook/Meta .... thanks to network effect I kinda still need it. I've managed to get some of my contacts over to telegram/signal but not everyone.</p>
WhatsApp worked but signed me out of the other phone (ffs). Bonus points for receiving an important message from someone I haven't heard from in about a year on the wrong phone. Ugh.</p>
So on the plus side I've learned of another reason to dislike WhatsApp.</p>
Hiding the unlock & root</h3>
Magisk has in its settings a "hide" option. This had no noticeable effect on the "SafteyNet" test (still three of three tests fail for me) built in to Magisk. I didn't notice any change in app behaiour in my limited poking.</p>
WiFi-based location (turn on)</h3>
Understandably from a freedom-warrior point of view the wifi-based loction services are off by default, but this is a bit of a pain because if you've ever used pure GPS positioning you'll know it's shit at doing it in houses, offices and built up areas, takes ages to get a lock, and is a bit hit and miss in cars. The accuracy you normally see is much more a function of the ubiquity of wifi in our urban lives.</p>
Screen refresh rate 60/120Hz (increase)</h3>
The "minimum screen refresh rate" setting was set to 60Hz. There's also a developer option for showing the current rate. In my experiment it sat at 60Hz all the time. This is fine till you scroll up and down, then you notice how much smoother the text scrolls at 120Hz. I upped the minimum to 120Hz which seemd to work.</p>
There's also a smooth-scroll option that I turned on that seems to produce a nicer effect.</p>
Using adb to do backups</h2>
It might be possible to backup a locked down phone without root by using adb's backup capabilities. I haven't looked into this much or tried it yet but here's some quick research notes:</p>

gist: Backup android app, data included, no root needed, with adb</a></li>
https://net2.com/how-to-backup-and-restore-your-android-device-with-adb-on-ubuntu/</a></li>
https://www.paulligocki.com/backup-restore-android-with-adb/</a></li>
https://stackpointer.io/mobile/android-adb-backup-extract-restore-repack/372/</a></li>
https://stackoverflow.com/questions/19225467/backing-up-android-device-using-adb</a></li>
https://android.stackexchange.com/questions/69567/what-all-does-adb-backup-and-how-do-i-restore-part-of-it</a></li>
</ul>
This might give me a way of rescuing some of my data from my old phone that de-rooted itself in an OS upgrade and fucked my titanium backups (still bitter).</p>
Oh but wait...</p>

Devs can set the ALLOW_BACKUP flag to "No"</a></p>
</blockquote>
Argh, what is wrong with this platform.</p>
Customisations</h2>

Turn on all the power button options:

Settings > system > advanced (ffs) > gestures (wtf?) > power menu > advanced restart [on]</li>
</ul>
</li>
</ul>
Options - what now?</h2>
That's been an enlightening exercise in "is this even possible". I've learned a huge amount about the current state of the phone ecosystem. I'm in equal measure dismayed and hopeful, there are some unpleasant trends and circumstances, and some really encouraging signs of hope for the open ecosystem.</p>
There really isn't a clear answer for me for where to go from here. There are downsides and challenges in every direction.</p>
The options available seem to be in increasing order of freedom and somewhat decreasing order of utility:</p>

Get an iPhone and abandon my principles and control, go live in a digital nursing home. - YeahNo.

get banking apps galore</li>
never get root again</li>
trust apple to administer the correct dosage of backups</li>
forever more fight the divide between iOS and my linux laptops</li>
</ul>
</li>
Install Oxygen 11, relock the bootloader, live without root, have no upgrade path because Oxygen is dead. - No.

possibly brick the phone trying to lock it</li>
get banking apps</li>
never get root again</li>
never have decent backups</li>
forever more fight the divide between iOS and my linux laptops</li>
</ul>
</li>
Install lineage and one of the google apps installers

still no bank</li>
probably get reliable google goodness</li>
get stalked by google (no change there, I dno't mind as much as some about this)</li>
unknown time-to-failure</li>
I think this is probably the most "mainstream" of the subversive options so is most likely to continue to have a forward path for the next few years</li>
worry about security of unlocked evenrything and the security of lineage - I think it's probably no worse than linux at this point but who knows</li>
</ul>
</li>
Install lineage-for-microG (current experiment)

no bank, no voice-to-text</li>
worry more about the security of build servers for this niche option</li>
I don't know how sustainable this project is, it's certainly noble</li>
</ul>
</li>
go all in and run a linux distro

definitely no apps for you</li>
get to take the moral high ground</li>
lose touch with everyone who is on whatsapp</li>
</ul>
</li>
</ul>
It would be nice to think that by choosing to run the most open thing I can I am in some way helping, but I think the truth is users really don't make much difference to projects like this unless they come in enough numbers to sway app vendors like WhatsCrapp to support more obscure platforms. It's only the hackers and those who can financially fund projects who really make a difference here. So while I want to run the most open thing I can it doesn't really matter a hill of beans outside of my own brain whether I do or not.</p>
I think having written down the options I'm swinging towards lineage + proprietary google services. This seems to solve the immediate problem of getting unlocked and rooted and getting working backups again without losing everything that's useful to me in the process.</p>
I'm glad I've tried the microG version, and hope we see open source slowly chip away at the power of the two giants just like Linux did in the desktop space over decades.</p>
Installing LineageOS + google services on OnePlus 9 Pro (take 2) - fail</h2>
I now have the lineage bootloader so the steps to get into recovery etc are a bit different than with stock or TWRP.</p>

"Note: If you want the Google Apps add-on on your device, you must follow this step before booting into LineageOS for the first time!" ~ https://wiki.lineageos.org/devices/lemonadep/install#installing-lineageos-from-recovery</a></p>
</blockquote>

Download latest nightly (there's no stable/unstable on this): https://download.lineageos.org/lemonadep</a> - this has both the ROM (OS image) and the recovery image.</li>
Download copy-partitions https://androidfilehost.com/?fid=2188818919693768129</a> (as per wiki)

The sha256 for the copy I have is 200877dfd0869a0e628955b807705765a91e34dff3bfeca9f828e916346aa85f  copy-partitions-20210323_1922.zip</code></li>
</ol>
</li>
Verify all the sha256 sums: sha256sum -c *.sha256</code></li>
Boot to fastboot</li>
flash the lineage recovery: fastboot flash boot lineage-18.1-20211228-recovery-lemonadep.img</code>

don't reboot yet</li>
</ol>
</li>
copy partitions as instructed

switch to recovery</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload copy-partitions-20210323_1922.zip</code></li>
Ignore unknown sig and "continue"</li>
come back out of recovery menu (don't reboot yet)</li>
don't reboot yet</li>
</ol>
</li>
"factory reset" > "wipe all data"

don't reboot yet</li>
</ol>
</li>
flash OS:

"apply update" > "apply from adb" (aka sideload)</li>
adb sideload lineage-18.1-20211228-nightly-lemonadep-signed.zip</code></li>
don't reboot yet</li>
</ol>
</li>
MindTheGapps - I haven't found any instructions for this anywhere

check the arch (it's arm64</code>) here https://wiki.lineageos.org/devices/lemonadep/</a></li>
Download the arm64 build from https://androidfilehost.com/?w=files&flid=322935</a>

or the "mirror" http://downloads.codefi.re/jdcteam/javelinanddart/gapps</a></li>
</ol>
</li>
Given there's no shas anywhere I downloaded from several mirrors and cross-checked the hashes. 85481cb98c8a8692f52c033ead1db436870af385498a917701fcd8c6182e145c  MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li>
ignore signature warning (the price of escaping a closed ecosystem)</li>
error shown on phone, roughly:low resource device detected, removing large extras</span></span>
not enough space for gapps! aborting</span></span>
...</span></span>
error in /sideload/package.zip (status 1)</span></span></code></pre>

patch that adds error message</a></li>
current message on sigma</code> branch</a></li>
In lineage recovery, mount sytsem</code> and enable adb, run df:OnePlus9Pro:/ # df -h /mnt/system                                                                                            </span></span>
Filesystem      Size  Used Avail Use% Mounted on</span></span>
/dev/block/dm-2 0.9G  0.9G  3.0M 100% /mnt/system</span></span></code></pre>That is indeed quite full. Annoyingly the installer doesn't say which partition is full.</li>
Some ideas here https://forum.xda-developers.com/t/q-insufficient-storage-space-in-system-partition.3018464/</a></li>
Try format & reboot to recovery, nope.</li>
hint that it could be slot related https://www.reddit.com/r/LineageOS/comments/fuykda/lineage_os_171_opengapps_error_not_sufficient/</a></li>
let's try a swap slot as per https://www.reddit.com/r/LineageOS/comments/fqashj/difficulty_installing_gapps_after_lineage_os/fluhd8s/?context=3</a></li>
enter fastboot</li>
</ol>
tim@max:~/Downloads/oneplus9pro/MindTheGapps</span></span>
$ fastboot set_active other</span></span>
Setting current slot to 'b'                        OKAY [  0.043s]</span></span>
Finished. Total time: 0.044s</span></span></code></pre>

enter recovery</li>
re-run the sideload</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li>
ignore signature warning (the price of escaping a closed ecosystem)</li>
same space error. sigh</li>
</ol>
</li>
</ol>
</li>
</ol>
Possible cause of space error</h3>

Partition not mounted at all</a></li>
LineageOS image has partitions that are too small</a> to fit extra g-apps in.

Maybe use TWRP BigSys</a> to get bigger partitions??</li>
Manually re-partition? Alegedly dangerous.

https://android.stackexchange.com/questions/216123/android-how-to-increase-system-partition-and-decrease-data-partition#comment276924_216123</a> warning of brickability if you touch wrong device</li>
https://www.reddit.com/r/Android/comments/2o8lvf/why_cant_we_resize_partitions_on_android_but_on/</a></li>
</ul>
</li>
</ul>
</li>
Incompatibility with Lineage bootloader - use TWRP instead</a> - worth a try I think, probably the next thing I'll try.</li>
Ruled Out: Wrong/stale slot in use</a> - tried swap slot (above), no change.</li>
Ruled out: Booting phone before gapps install (didn't do this so not this).</li>
</ol>
Patching the installer - abandoned</h3>
It seems to me having read around that the error message is hiding useful information on the state of the device and what exactly failed. (Which partition is out of space? Is it just read only or something?). I'n not feeling like playing with partitioning given the warnings I saw on a forum of potentially bricking a device</p>
There's links to the source above and I did start fiddling with this, but you have to figure out how to re-sign the zip to get it to install at which point I fell down an android-sized rabbit hole.</p>
Signing zips and apks</h4>
Here's what I found so far, I haven't got it working yet.</p>
I think</em> it's the same signing that you have to do to apk</code> files, and apk</code> is just a special shaped zip</code> file. Off into learning-android-sdk-land I went...</p>

There's some kind of strange android app that can sign zips called ZipSigner</a>

ZipSigner is missing source code</a> so you are relying on random binaries. Dodgy but probably well intentioned.</li>
</ul>
</li>
https://android.stackexchange.com/questions/222262/how-to-create-manually-create-my-own-%e1%b4%8f%e1%b4%9b%e1%b4%80-update-file-to-be-used-for-adb-sideload</a></li>
https://android.stackexchange.com/questions/95425/update-zip-just-for-fixing-file-permissions-possible</a></li>
https://forum.xda-developers.com/t/tutorial-the-updater-script-completely-explained.2377695/</a></li>
https://www.bettermobileapp.com/article/10558021/Updater-script</a></li>
https://android.stackexchange.com/questions/191043/edditing-updater-script</a></li>
https://androidforums.com/threads/custom-rom-updater-scipt-troubleshooting.701752/</a></li>
https://android.stackexchange.com/questions/35600/how-to-create-an-update-zip-file-that-can-copy-rename-a-file</a></li>
https://developer.android.com/studio/command-line/zipalign</a></li>
https://developer.android.com/studio/publish/app-signing#opt-out</a></li>
https://developer.android.com/studio/command-line/apksigner</a></li>
https://www.addictivetips.com/mobile/what-is-zipalign-in-android-and-how-it-works-complete-guide/</a>

https://stackoverflow.com/questions/3994035/what-is-aligned-memory-allocation</a></li>
</ul>
</li>
https://www.androidcentral.com/installing-android-sdk-windows-mac-and-linux-tutorial</a></li>
</ul>
Let's try TWRP+Lineage+MindTheGapps - fail</h3>
Install TWRP</h4>
https://www.getdroidtips.com/replace-lineageos-recovery-twrp/</a></p>

Reboot phone to fastboot or something (the built-in one where the volume key changes options), leave it in that state</li>
fastboot devices</code> on linux should show the phone</li>
fastboot boot twrp.img</code> (already downloaded and symlinked above)</li>
Phone reboots into twrp temporarily</li>
Advanced > flash current twrp</li>
Reboot (ok the no-OS warning)</li>
Yay, booted into twrp, comforting</li>
</ol>
Install LineageOS (yet again)</h4>

Well, let's get a new nightly</li>
Instructions say check model...

Reboot to twrp (recovery) with USB connected to laptop</li>
Laptop pops window for new mount mtp://OnePlus_LE2123_dade278d/</code></li>
Yep, LE2123 is in the supported list. Phew.</li>
</ul>
</li>
Skip all the "recovery" steps in https://wiki.lineageos.org/devices/lemonadep/install</a></li>
I'm guessing I don't need to run "copy partitions" again because that's to avoid only having one working slot, and I already have to having done this before. Tell me if I'm wrong!</li>
Give up waiting for new image download (~1GB), continue with older image...</li>
Factory reset

TWRP > Wipe > Swipe for factory reset</li>
back</li>
Format Data (button)</li>
type yes to continue (wiping all apps & data)</li>
back to main menu (android back button lots of times)</li>
</ul>
</li>
Flash lineage

Still in TWRP</li>
Advanced > ADB Sideload</li>
Tick the wipe boxes ("Dalvik cache" + "cache")</li>
Swipe to sart sideload</li>
adb sideload lineage-18.1-20211228-nightly-lemonadep-signed.zip</code></li>
back</li>
Don't reboot yet</li>
</ul>
</li>
Add MindTheGaps

still in TWRP</li>
ADB Sideload</li>
swipe (cache wipe still selected)</li>
adb sideload MindTheGapps-11.0.0-arm64-20210920_084011.zip</code></li>
fail. log:</li>
</ul>
</li>
</ul>
**********************</span></span>
MindTheGapps installer</span></span>
**********************</span></span>
Extracting files</span></span>
Setting up environment</span></span>
Mounting partitions</span></span>
Could not mount /mnt/system! Aborting</span></span>
Updater process ended with ERROR: 1</span></span></code></pre>
For goodness sake.</p>
Fuck it, let's try the other gapps even though it says not to....</p>
Reboot just to have a look... Lineage running fine, no sign of any gapps.</p>
Let's try TWRP+Lineage+OpenGApps - fail</h3>

Power off</li>
power + vol-down - ended up in lineage recovery, wat?? swap slot??</li>
reboot to bootloader? no that's the built-in pixelated bright green "START" thing, oh actually that's fine, that's the fastboot thing.</li>
switch slot as above</li>
</ul>
fastboot set_active other</span></span>
Setting current slot to 'b'</span></span>
... OKAY [  0.048s]</span></span>
Finished. Total time: 0.056s</span></span></code></pre>

reboot to recovery</li>
back in TWRP</li>
</ul>
That's cool! I have lineage recovery in slot a</code> and TWRP in slot b</code>.</p>
Okay, back to the plan.</p>
Lineage (again again again...)</h4>

Repeat the lineage intall steps above (no idea if that's really necessary but shrug</em>)</li>
</ul>
OpenGApps install - fail</h4>

Download https://opengapps.org/?api=11.0&variant=nano</a> - ARM64 / 11 / nano

(manually changed from 10 to 11 from link on https://wiki.lineageos.org/gapps</a>)</li>
download the md5 too</li>
md5sum -c *.md5</code> to validate...</li>
</ul>
</li>
Back to ADB Sideload in TWRP (cache wipe ticked x2), swipe to start</li>
adb sideload open_gapps-arm64-11.0-nano-20220215.zip</code></li>
cross fingers</li>
bah. failed, basically same error.</li>
</ul>
- Performing system space calculations</span></span>
</span>
Insufficient storage space available in</span></span>
System partition. ...</span></span>
...</span></span>
Error Code: 70</span></span></code></pre>
grap logs as instructed: adb pull /sdcard/open_gapps_log.txt</code> - success</p>
gist of log file open_gapps_log.txt</a></p>
Important bit here is that there is ZERO</strong> space, so choosing smaller install is not going to work.</p>
...</span></span>
Current Free Space |         0</span></span>
...</span></span></code></pre>
Well on the plus side the diagnostics are muuuuch better in OpenGApps so that's saved me fucking around trying to patch MindTheGaps' installer.</p>
Two-phone-Timmy - the burner phone</h2>
Having spent some time completely stumped by the space issue and giving it some thought I realize that this battle with Android will never end. Trying to get an unholy mix of Libre and proprietary to mix is like oil and water.</p>
I can opt out of that battle by running two phones.</p>

A "primary" phone that I use as much as possible, running the Free-est thing I can muster. This would get the sim card. This would be the OnePlus 9 pro, ideally running PureOS as per the Librem 5 phone, but if not that then Lineage without the GApps that I couldn't get to work.</li>
A burner phonee that's basically stock Android so that I can fall back to this for everything that can't or won't work on a free-er & unlocked base OS. (My existing OnePlus 5t for now while it lasts, reset to stock OxygenOS). This one can the get internet by tethering to the primary. I don't think any of the apps actually require the sim to be present. I guess we'll see.</li>
</ol>
The primary (9 Pro)</h3>
The idea here is to get the free-est possible base layer (i.e. operating system), and then get maximum functionality within that. Anything that doesn't work or I can't find a working alternative to gets punted to the burner phone while we wait for hell to freeze over again. This will be the compromise for now.</p>

Research trying PureOS, see what the app ecosystem is like - done - nope

research done, doesn't seem to be possible to install on the OnePlus. Next...</li>
</ul>
</li>
Continue with Lineage (I left the phone with a working but empty install of this)</li>
Don't install proprietary google play services or play store at all.</strong> - This is the line in the sand. I would have accepted it for now, but given I couldn't install it, I'll take it as the new battle line for free vs proprietary.</li>
Install microG for re-implementation of shared services (i.e. location, polling for updates from servers etc)</li>
Install Aurora play store proxy again to be able to try out app store apps</li>
See how close I can get to a usable phone.</li>
</ul>
The burner (5t)</h3>

Factory reset the 5t</li>
Install just what I couldn't get to work on the primary</li>
Cry because the backups were shit and now I've lost access to something I care about (I predict)</li>
</ul>
Primary setup - experimenting</h2>
Fired up the lineage (minus all google stuff) that I'd installed and then failed to get any gapps on (space issue above, no longer care).</p>
Went through language prompt, was looking at time prompt in setup wizard when the phone rebooted with some kind of couldn't read data error. Disturbing. Rebooted, selected "try again" and it didn't do it this time.</p>
Setup wizard</h3>
- selected UK location,</span></span>
- GMT timezone.</span></span>
- Put in wifi password (incorrectly)</span></span>
- set up pin</span></span>
- set up fingerprint</span></span>
- booted into OS (yay, pleasing to see Lineage in action)</span></span></code></pre>OS update with on-phone updater</h3>
settings > advanced > updater</p>
shows list of OS updates. hit refresh, nothing new. latest is 18.1 26th Apr 2022</p>
tap download ....</p>
tap intall (even longer pause ... installing ... finalising package installation ...) ..</p>
reboot (press button that's presented)</p>
config</h3>

turn off annoying sounds

settings > sound > advanced</li>
screen locking sound - off</li>
touch sounds - off</li>
</ul>
</li>
</ul>
todo</h3>

replacement contact management (currently using google)</li>
replacement calendar management (currently using google)</li>
insert sim</li>
</ul>
already google-free</h3>

email (fastmail domain + k9mail app)</li>
broswer - firefox</li>
search - duckduckgo</li>
OTP - andOTP</li>
maps/satnav (partially) - could use waze + maps.me (google maps is a decent car satnav, not found a perfect equivalent)</li>
</ul>
burner phone</h2>
No sim in this one, use wifi and/or bluetooth tethering to get internet on it.</p>
Things I can't escape the googly jail for</p>

google maps?</li>
apps for clients

microsoft outlook</li>
securid (vpn stuff)</li>
google authenticator (use andOTP instead wherever possible)</li>
microsoft authenticator (yuck, what's wrong with the OTP standard?!?!)</li>
digidentity</li>
</ul>
</li>
banking apps that don't work on the primary phone (not naming because haxxors)</li>
google pay (never used it, just use an actual card)</li>
google home for controlling the chromecast</li>
spotify just for casting to the chromecast audio</li>
</ul>
To my horror I have discovered that android allows apps to force remote admin privs so that IT departments can snoop on your BYOD. Hello Microsoft Outlook + Active Directory. Yet more erosion of user rights in the name of security. There's no way in hell I'm allowing a client to be an admin on my primary phone so this is burner phone territory if at all necessary; though I just won't install any of that junk at all and will be uncontactable within their network unless on their supplied devices. Shrug.</p>
Primary phone - microG</h2>
lineage+microG - failed install</h3>
Re-run setup in section: Installing “Lineage for microG”</p>

OS install

power off</li>
boot into lineage recovery (vol-down + power) (currently slot "b" is active)</li>
"apply update" > "apply from adb" (aka sideload)</li>
adb sideload lineage-18.1-20211220-microG-lemonadep.zip</code>

it seems this is flashed to slot a and then slot is automatically swapped</li>
</ul>
</li>
ignore signature warning</li>
accept downgrade warning</li>
completed</li>
back back</li>
factory reset</li>
reboot</li>
drops to bootloader thing</li>
"start" (power button to accept)</li>
loops back to bootloader thing. damn.</li>
vol down vol down - Recovery - power button to accept</li>
starts lineage recovery - "Active slot: a" - hmmm</li>
</ul>
</li>
</ul>
try swap slot</p>
$ fastboot set_active other</span></span>
Setting current slot to 'b'                        (bootloader) Changing the active slot with a snapshot applied may cancel the update.</span></span>
OKAY [  0.038s]</span></span>
Finished. Total time: 0.040s</span></span></code></pre>
power off, power on</p>
that worked, now enters the lineage setup steps</p>
oh but it's back to the blank OS, the apr 2022 version!.</p>
guess there's a slot for two OSes!</p>
swap slot again, back to the bootloader loop. interesting.</p>
let's try installing again</p>
fastboot set_active other</span></span>
Setting current slot to 'a'                        OKAY [  0.060s]</span></span>
Finished. Total time: 0.061s</span></span></code></pre>tim@max:~/Downloads/oneplus9pro/lineage-for-microG      </span></span>
$ adb sideload lineage-18.1-20211220-microG-lemonadep.zip                                                                     </span></span>
serving: 'lineage-18.1-20211220-microG-lemonadep.zip'  (~21%)</span></span></code></pre>
factory reset</p>
reboot</p>
cross fingers. nope. still just drops back to bootloader</p>
recovery > factory reset > reboot</p>
nope still fooked
reboot to bootloader which says "Active slot: b" which is a bit odd. Maybe it does a swap slot automatically after flashing?</p>
so at this point slot a has a fooked lineage-microG (maybe) and slot b has a functioning but blank vanilla lineage. I think.</p>
https://www.reddit.com/r/GooglePixel/comments/8cz6m9/google_pixel_bootslot_b_brick/</a></p>
grab the vars with fastboot getvar all 2>&1 | sort > fastboot-getvar.txt</code></p>
output (sensitive things redacted with XXXX) - interesting but not super helpful:</p>
all: </span></span>
(bootloader) battery-voltage:0</span></span>
(bootloader) cpu-abi:arm64-v8a</span></span>
(bootloader) current-slot:b</span></span>
(bootloader) dynamic-partition:true</span></span>
(bootloader) first-api-level:30</span></span>
(bootloader) has-slot:abl_log:no</span></span>
(bootloader) has-slot:abl:yes</span></span>
(bootloader) has-slot:ALIGN_TO_128K_1:no</span></span>
(bootloader) has-slot:ALIGN_TO_128K_2:no</span></span>
(bootloader) has-slot:android_log:no</span></span>
(bootloader) has-slot:aop:yes</span></span>
(bootloader) has-slot:apdp_full:no</span></span>
(bootloader) has-slot:apdp:no</span></span>
(bootloader) has-slot:bluetooth:yes</span></span>
(bootloader) has-slot:boot:yes</span></span>
(bootloader) has-slot:carrier:no</span></span>
(bootloader) has-slot:cdt:no</span></span>
(bootloader) has-slot:connsec:no</span></span>
(bootloader) has-slot:cpucp:yes</span></span>
(bootloader) has-slot:ddr:no</span></span>
(bootloader) has-slot:devcfg:yes</span></span>
(bootloader) has-slot:devinfo:no</span></span>
(bootloader) has-slot:dinfo:no</span></span>
(bootloader) has-slot:dip:no</span></span>
(bootloader) has-slot:DRIVER:no</span></span>
(bootloader) has-slot:dsp:yes</span></span>
(bootloader) has-slot:dtbo:yes</span></span>
(bootloader) has-slot:engineering_cdt:yes</span></span>
(bootloader) has-slot:featenabler:yes</span></span>
(bootloader) has-slot:frp:no</span></span>
(bootloader) has-slot:fsc:no</span></span>
(bootloader) has-slot:fsg:no</span></span>
(bootloader) has-slot:hyp_log:no</span></span>
(bootloader) has-slot:hyp:yes</span></span>
(bootloader) has-slot:imagefv:yes</span></span>
(bootloader) has-slot:kernel_log:no</span></span>
(bootloader) has-slot:keymaster:yes</span></span>
(bootloader) has-slot:keystore:no</span></span>
(bootloader) has-slot:limits-cdsp:no</span></span>
(bootloader) has-slot:limits:no</span></span>
(bootloader) has-slot:logdump:no</span></span>
(bootloader) has-slot:logfs:no</span></span>
(bootloader) has-slot:mdcompress:no</span></span>
(bootloader) has-slot:mdtpsecapp:yes</span></span>
(bootloader) has-slot:mdtp:yes</span></span>
(bootloader) has-slot:metadata:no</span></span>
(bootloader) has-slot:misc:no</span></span>
(bootloader) has-slot:modemdump:no</span></span>
(bootloader) has-slot:modemst1:no</span></span>
(bootloader) has-slot:modemst2:no</span></span>
(bootloader) has-slot:modem:yes</span></span>
(bootloader) has-slot:multiimgoem:yes</span></span>
(bootloader) has-slot:ocdt:no</span></span>
(bootloader) has-slot:odm_b-cow:no</span></span>
(bootloader) has-slot:odm:yes</span></span>
(bootloader) has-slot:oplusdycnvbk:no</span></span>
(bootloader) has-slot:opluslog:no</span></span>
(bootloader) has-slot:oplusreserve1:no</span></span>
(bootloader) has-slot:oplusreserve2:no</span></span>
(bootloader) has-slot:oplusreserve3:no</span></span>
(bootloader) has-slot:oplusreserve4:no</span></span>
(bootloader) has-slot:oplusreserve5:no</span></span>
(bootloader) has-slot:oplus_sec:yes</span></span>
(bootloader) has-slot:oplusstanvbk:yes</span></span>
(bootloader) has-slot:param:no</span></span>
(bootloader) has-slot:persist_bkp:no</span></span>
(bootloader) has-slot:persist:no</span></span>
(bootloader) has-slot:product_b-cow:no</span></span>
(bootloader) has-slot:product:yes</span></span>
(bootloader) has-slot:qmcs:no</span></span>
(bootloader) has-slot:qsee_log:no</span></span>
(bootloader) has-slot:qupfw:yes</span></span>
(bootloader) has-slot:qweslicstore:yes</span></span>
(bootloader) has-slot:rawdump:no</span></span>
(bootloader) has-slot:rtice:no</span></span>
(bootloader) has-slot:sda:no</span></span>
(bootloader) has-slot:sdb:no</span></span>
(bootloader) has-slot:sdc:no</span></span>
(bootloader) has-slot:sdd:no</span></span>
(bootloader) has-slot:sde:no</span></span>
(bootloader) has-slot:sdf:no</span></span>
(bootloader) has-slot:secdata:no</span></span>
(bootloader) has-slot:shrm:yes</span></span>
(bootloader) has-slot:splash:yes</span></span>
(bootloader) has-slot:spunvm:no</span></span>
(bootloader) has-slot:ssd:no</span></span>
(bootloader) has-slot:storsec:no</span></span>
(bootloader) has-slot:super:no</span></span>
(bootloader) has-slot:system_b-cow:no</span></span>
(bootloader) has-slot:system_ext_b-cow:no</span></span>
(bootloader) has-slot:system_ext:yes</span></span>
(bootloader) has-slot:system:yes</span></span>
(bootloader) has-slot:tzsc:no</span></span>
(bootloader) has-slot:tz:yes</span></span>
(bootloader) has-slot:uefisecapp:yes</span></span>
(bootloader) has-slot:uefivarstore:no</span></span>
(bootloader) has-slot:userdata:no</span></span>
(bootloader) has-slot:vbmeta_system:yes</span></span>
(bootloader) has-slot:vbmeta_vendor:yes</span></span>
(bootloader) has-slot:vbmeta:yes</span></span>
(bootloader) has-slot:vendor_b-cow:no</span></span>
(bootloader) has-slot:vendor_boot:yes</span></span>
(bootloader) has-slot:vendor:yes</span></span>
(bootloader) has-slot:vm-bootsys:yes</span></span>
(bootloader) has-slot:vm-data:no</span></span>
(bootloader) has-slot:xbl_config:yes</span></span>
(bootloader) has-slot:xbl:yes</span></span>
(bootloader) hw-revision:0</span></span>
(bootloader) is-logical:abl_a:no</span></span>
(bootloader) is-logical:abl_b:no</span></span>
(bootloader) is-logical:abl_log:no</span></span>
(bootloader) is-logical:ALIGN_TO_128K_1:no</span></span>
(bootloader) is-logical:ALIGN_TO_128K_2:no</span></span>
(bootloader) is-logical:android_log:no</span></span>
(bootloader) is-logical:aop_a:no</span></span>
(bootloader) is-logical:aop_b:no</span></span>
(bootloader) is-logical:apdp_full:no</span></span>
(bootloader) is-logical:apdp:no</span></span>
(bootloader) is-logical:bluetooth_a:no</span></span>
(bootloader) is-logical:bluetooth_b:no</span></span>
(bootloader) is-logical:boot_a:no</span></span>
(bootloader) is-logical:boot_b:no</span></span>
(bootloader) is-logical:carrier:no</span></span>
(bootloader) is-logical:cdt:no</span></span>
(bootloader) is-logical:connsec:no</span></span>
(bootloader) is-logical:cpucp_a:no</span></span>
(bootloader) is-logical:cpucp_b:no</span></span>
(bootloader) is-logical:ddr:no</span></span>
(bootloader) is-logical:devcfg_a:no</span></span>
(bootloader) is-logical:devcfg_b:no</span></span>
(bootloader) is-logical:devinfo:no</span></span>
(bootloader) is-logical:dinfo:no</span></span>
(bootloader) is-logical:dip:no</span></span>
(bootloader) is-logical:DRIVER:no</span></span>
(bootloader) is-logical:dsp_a:no</span></span>
(bootloader) is-logical:dsp_b:no</span></span>
(bootloader) is-logical:dtbo_a:no</span></span>
(bootloader) is-logical:dtbo_b:no</span></span>
(bootloader) is-logical:engineering_cdt_a:no</span></span>
(bootloader) is-logical:engineering_cdt_b:no</span></span>
(bootloader) is-logical:featenabler_a:no</span></span>
(bootloader) is-logical:featenabler_b:no</span></span>
(bootloader) is-logical:frp:no</span></span>
(bootloader) is-logical:fsc:no</span></span>
(bootloader) is-logical:fsg:no</span></span>
(bootloader) is-logical:hyp_a:no</span></span>
(bootloader) is-logical:hyp_b:no</span></span>
(bootloader) is-logical:hyp_log:no</span></span>
(bootloader) is-logical:imagefv_a:no</span></span>
(bootloader) is-logical:imagefv_b:no</span></span>
(bootloader) is-logical:kernel_log:no</span></span>
(bootloader) is-logical:keymaster_a:no</span></span>
(bootloader) is-logical:keymaster_b:no</span></span>
(bootloader) is-logical:keystore:no</span></span>
(bootloader) is-logical:limits-cdsp:no</span></span>
(bootloader) is-logical:limits:no</span></span>
(bootloader) is-logical:logdump:no</span></span>
(bootloader) is-logical:logfs:no</span></span>
(bootloader) is-logical:mdcompress:no</span></span>
(bootloader) is-logical:mdtp_a:no</span></span>
(bootloader) is-logical:mdtp_b:no</span></span>
(bootloader) is-logical:mdtpsecapp_a:no</span></span>
(bootloader) is-logical:mdtpsecapp_b:no</span></span>
(bootloader) is-logical:metadata:no</span></span>
(bootloader) is-logical:misc:no</span></span>
(bootloader) is-logical:modem_a:no</span></span>
(bootloader) is-logical:modem_b:no</span></span>
(bootloader) is-logical:modemdump:no</span></span>
(bootloader) is-logical:modemst1:no</span></span>
(bootloader) is-logical:modemst2:no</span></span>
(bootloader) is-logical:multiimgoem_a:no</span></span>
(bootloader) is-logical:multiimgoem_b:no</span></span>
(bootloader) is-logical:ocdt:no</span></span>
(bootloader) is-logical:odm_b-cow:yes</span></span>
(bootloader) is-logical:odm_b:yes</span></span>
(bootloader) is-logical:oplusdycnvbk:no</span></span>
(bootloader) is-logical:opluslog:no</span></span>
(bootloader) is-logical:oplusreserve1:no</span></span>
(bootloader) is-logical:oplusreserve2:no</span></span>
(bootloader) is-logical:oplusreserve3:no</span></span>
(bootloader) is-logical:oplusreserve4:no</span></span>
(bootloader) is-logical:oplusreserve5:no</span></span>
(bootloader) is-logical:oplus_sec_a:no</span></span>
(bootloader) is-logical:oplus_sec_b:no</span></span>
(bootloader) is-logical:oplusstanvbk_a:no</span></span>
(bootloader) is-logical:oplusstanvbk_b:no</span></span>
(bootloader) is-logical:param:no</span></span>
(bootloader) is-logical:persist_bkp:no</span></span>
(bootloader) is-logical:persist:no</span></span>
(bootloader) is-logical:product_b-cow:yes</span></span>
(bootloader) is-logical:product_b:yes</span></span>
(bootloader) is-logical:qmcs:no</span></span>
(bootloader) is-logical:qsee_log:no</span></span>
(bootloader) is-logical:qupfw_a:no</span></span>
(bootloader) is-logical:qupfw_b:no</span></span>
(bootloader) is-logical:qweslicstore_a:no</span></span>
(bootloader) is-logical:qweslicstore_b:no</span></span>
(bootloader) is-logical:rawdump:no</span></span>
(bootloader) is-logical:rtice:no</span></span>
(bootloader) is-logical:sda:no</span></span>
(bootloader) is-logical:sdb:no</span></span>
(bootloader) is-logical:sdc:no</span></span>
(bootloader) is-logical:sdd:no</span></span>
(bootloader) is-logical:sde:no</span></span>
(bootloader) is-logical:sdf:no</span></span>
(bootloader) is-logical:secdata:no</span></span>
(bootloader) is-logical:shrm_a:no</span></span>
(bootloader) is-logical:shrm_b:no</span></span>
(bootloader) is-logical:splash_a:no</span></span>
(bootloader) is-logical:splash_b:no</span></span>
(bootloader) is-logical:spunvm:no</span></span>
(bootloader) is-logical:ssd:no</span></span>
(bootloader) is-logical:storsec:no</span></span>
(bootloader) is-logical:super:no</span></span>
(bootloader) is-logical:system_b-cow:yes</span></span>
(bootloader) is-logical:system_b:yes</span></span>
(bootloader) is-logical:system_ext_b-cow:yes</span></span>
(bootloader) is-logical:system_ext_b:yes</span></span>
(bootloader) is-logical:tz_a:no</span></span>
(bootloader) is-logical:tz_b:no</span></span>
(bootloader) is-logical:tzsc:no</span></span>
(bootloader) is-logical:uefisecapp_a:no</span></span>
(bootloader) is-logical:uefisecapp_b:no</span></span>
(bootloader) is-logical:uefivarstore:no</span></span>
(bootloader) is-logical:userdata:no</span></span>
(bootloader) is-logical:vbmeta_a:no</span></span>
(bootloader) is-logical:vbmeta_b:no</span></span>
(bootloader) is-logical:vbmeta_system_a:no</span></span>
(bootloader) is-logical:vbmeta_system_b:no</span></span>
(bootloader) is-logical:vbmeta_vendor_a:no</span></span>
(bootloader) is-logical:vbmeta_vendor_b:no</span></span>
(bootloader) is-logical:vendor_b-cow:yes</span></span>
(bootloader) is-logical:vendor_boot_a:no</span></span>
(bootloader) is-logical:vendor_boot_b:no</span></span>
(bootloader) is-logical:vendor_b:yes</span></span>
(bootloader) is-logical:vm-bootsys_a:no</span></span>
(bootloader) is-logical:vm-bootsys_b:no</span></span>
(bootloader) is-logical:vm-data:no</span></span>
(bootloader) is-logical:xbl_a:no</span></span>
(bootloader) is-logical:xbl_b:no</span></span>
(bootloader) is-logical:xbl_config_a:no</span></span>
(bootloader) is-logical:xbl_config_b:no</span></span>
(bootloader) is-userspace:yes</span></span>
(bootloader) max-download-size:0x10000000</span></span>
(bootloader) partition-size:abl_a:0x800000</span></span>
(bootloader) partition-size:abl_b:0x800000</span></span>
(bootloader) partition-size:abl_log:0xFD0000</span></span>
(bootloader) partition-size:ALIGN_TO_128K_1:0x1A000</span></span>
(bootloader) partition-size:ALIGN_TO_128K_2:0x1A000</span></span>
(bootloader) partition-size:android_log:0xFD0000</span></span>
(bootloader) partition-size:aop_a:0x80000</span></span>
(bootloader) partition-size:aop_b:0x80000</span></span>
(bootloader) partition-size:apdp:0x40000</span></span>
(bootloader) partition-size:apdp_full:0x40000</span></span>
(bootloader) partition-size:bluetooth_a:0x200000</span></span>
(bootloader) partition-size:bluetooth_b:0x200000</span></span>
(bootloader) partition-size:boot_a:0xC000000</span></span>
(bootloader) partition-size:boot_b:0xC000000</span></span>
(bootloader) partition-size:carrier:0x4000000</span></span>
(bootloader) partition-size:cdt:0x20000</span></span>
(bootloader) partition-size:connsec:0x20000</span></span>
(bootloader) partition-size:cpucp_a:0x100000</span></span>
(bootloader) partition-size:cpucp_b:0x100000</span></span>
(bootloader) partition-size:ddr:0x100000</span></span>
(bootloader) partition-size:devcfg_a:0x20000</span></span>
(bootloader) partition-size:devcfg_b:0x20000</span></span>
(bootloader) partition-size:devinfo:0x1000</span></span>
(bootloader) partition-size:dinfo:0x4000</span></span>
(bootloader) partition-size:dip:0x100000</span></span>
(bootloader) partition-size:DRIVER:0x2000000</span></span>
(bootloader) partition-size:dsp_a:0x4000000</span></span>
(bootloader) partition-size:dsp_b:0x4000000</span></span>
(bootloader) partition-size:dtbo_a:0x1800000</span></span>
(bootloader) partition-size:dtbo_b:0x1800000</span></span>
(bootloader) partition-size:engineering_cdt_a:0x100000</span></span>
(bootloader) partition-size:engineering_cdt_b:0x100000</span></span>
(bootloader) partition-size:featenabler_a:0x20000</span></span>
(bootloader) partition-size:featenabler_b:0x20000</span></span>
(bootloader) partition-size:frp:0x80000</span></span>
(bootloader) partition-size:fsc:0x20000</span></span>
(bootloader) partition-size:fsg:0x300000</span></span>
(bootloader) partition-size:hyp_a:0x800000</span></span>
(bootloader) partition-size:hyp_b:0x800000</span></span>
(bootloader) partition-size:hyp_log:0xFD0000</span></span>
(bootloader) partition-size:imagefv_a:0x200000</span></span>
(bootloader) partition-size:imagefv_b:0x200000</span></span>
(bootloader) partition-size:kernel_log:0xFD0000</span></span>
(bootloader) partition-size:keymaster_a:0x80000</span></span>
(bootloader) partition-size:keymaster_b:0x80000</span></span>
(bootloader) partition-size:keystore:0x80000</span></span>
(bootloader) partition-size:limits:0x1000</span></span>
(bootloader) partition-size:limits-cdsp:0x1000</span></span>
(bootloader) partition-size:logdump:0x4000000</span></span>
(bootloader) partition-size:logfs:0x800000</span></span>
(bootloader) partition-size:mdcompress:0x1400000</span></span>
(bootloader) partition-size:mdtp_a:0x2000000</span></span>
(bootloader) partition-size:mdtp_b:0x2000000</span></span>
(bootloader) partition-size:mdtpsecapp_a:0x400000</span></span>
(bootloader) partition-size:mdtpsecapp_b:0x400000</span></span>
(bootloader) partition-size:metadata:0x1000000</span></span>
(bootloader) partition-size:misc:0x100000</span></span>
(bootloader) partition-size:modem_a:0x14A00000</span></span>
(bootloader) partition-size:modem_b:0x14A00000</span></span>
(bootloader) partition-size:modemdump:0x12C00000</span></span>
(bootloader) partition-size:modemst1:0x300000</span></span>
(bootloader) partition-size:modemst2:0x300000</span></span>
(bootloader) partition-size:multiimgoem_a:0x8000</span></span>
(bootloader) partition-size:multiimgoem_b:0x8000</span></span>
(bootloader) partition-size:ocdt:0x20000</span></span>
(bootloader) partition-size:odm_b:0x2BD000</span></span>
(bootloader) partition-size:odm_b-cow:0x2C1000</span></span>
(bootloader) partition-size:oplusdycnvbk:0xA00000</span></span>
(bootloader) partition-size:opluslog:0x10000000</span></span>
(bootloader) partition-size:oplusreserve1:0x800000</span></span>
(bootloader) partition-size:oplusreserve2:0x10000000</span></span>
(bootloader) partition-size:oplusreserve3:0x4000000</span></span>
(bootloader) partition-size:oplusreserve4:0x2000000</span></span>
(bootloader) partition-size:oplusreserve5:0x4000000</span></span>
(bootloader) partition-size:oplus_sec_a:0xA00000</span></span>
(bootloader) partition-size:oplus_sec_b:0xA00000</span></span>
(bootloader) partition-size:oplusstanvbk_a:0xA00000</span></span>
(bootloader) partition-size:oplusstanvbk_b:0xA00000</span></span>
(bootloader) partition-size:param:0x800000</span></span>
(bootloader) partition-size:persist:0x2000000</span></span>
(bootloader) partition-size:persist_bkp:0x2000000</span></span>
(bootloader) partition-size:product_b:0x1652F000</span></span>
(bootloader) partition-size:product_b-cow:0x16696000</span></span>
(bootloader) partition-size:qmcs:0x1E00000</span></span>
(bootloader) partition-size:qsee_log:0xFD0000</span></span>
(bootloader) partition-size:qupfw_a:0x14000</span></span>
(bootloader) partition-size:qupfw_b:0x14000</span></span>
(bootloader) partition-size:qweslicstore_a:0x40000</span></span>
(bootloader) partition-size:qweslicstore_b:0x40000</span></span>
(bootloader) partition-size:rawdump:0x8000000</span></span>
(bootloader) partition-size:rtice:0x80000</span></span>
(bootloader) partition-size:sda:0x3A10800000</span></span>
(bootloader) partition-size:sdb:0x800000</span></span>
(bootloader) partition-size:sdc:0x800000</span></span>
(bootloader) partition-size:sdd:0x2000000</span></span>
(bootloader) partition-size:sde:0x180000000</span></span>
(bootloader) partition-size:sdf:0x2000000</span></span>
(bootloader) partition-size:secdata:0x7000</span></span>
(bootloader) partition-size:shrm_a:0x20000</span></span>
(bootloader) partition-size:shrm_b:0x20000</span></span>
(bootloader) partition-size:splash_a:0x20A4000</span></span>
(bootloader) partition-size:splash_b:0x20A4000</span></span>
(bootloader) partition-size:spunvm:0x2000000</span></span>
(bootloader) partition-size:ssd:0x2000</span></span>
(bootloader) partition-size:storsec:0x20000</span></span>
(bootloader) partition-size:super:0x29B000000</span></span>
(bootloader) partition-size:system_b:0x406A4000</span></span>
(bootloader) partition-size:system_b-cow:0x40AAC000</span></span>
(bootloader) partition-size:system_ext_b:0xDF3A000</span></span>
(bootloader) partition-size:system_ext_b-cow:0xE01B000</span></span>
(bootloader) partition-size:tz_a:0x400000</span></span>
(bootloader) partition-size:tz_b:0x400000</span></span>
(bootloader) partition-size:tzsc:0x20000</span></span>
(bootloader) partition-size:uefisecapp_a:0x200000</span></span>
(bootloader) partition-size:uefisecapp_b:0x200000</span></span>
(bootloader) partition-size:uefivarstore:0x80000</span></span>
(bootloader) partition-size:userdata:0x3751FB3000</span></span>
(bootloader) partition-size:vbmeta_a:0x10000</span></span>
(bootloader) partition-size:vbmeta_b:0x10000</span></span>
(bootloader) partition-size:vbmeta_system_a:0x10000</span></span>
(bootloader) partition-size:vbmeta_system_b:0x10000</span></span>
(bootloader) partition-size:vbmeta_vendor_a:0x10000</span></span>
(bootloader) partition-size:vbmeta_vendor_b:0x10000</span></span>
(bootloader) partition-size:vendor_b:0x50520000</span></span>
(bootloader) partition-size:vendor_b-cow:0x50A27000</span></span>
(bootloader) partition-size:vendor_boot_a:0xC000000</span></span>
(bootloader) partition-size:vendor_boot_b:0xC000000</span></span>
(bootloader) partition-size:vm-bootsys_a:0x10625000</span></span>
(bootloader) partition-size:vm-bootsys_b:0x10625000</span></span>
(bootloader) partition-size:vm-data:0x20A4000</span></span>
(bootloader) partition-size:xbl_a:0x600000</span></span>
(bootloader) partition-size:xbl_b:0x600000</span></span>
(bootloader) partition-size:xbl_config_a:0x80000</span></span>
(bootloader) partition-size:xbl_config_b:0x80000</span></span>
(bootloader) product:OnePlus9Pro</span></span>
(bootloader) secure:yes</span></span>
(bootloader) security-patch-level:2021-12-05</span></span>
(bootloader) serialno:XXXXXXX</span></span>
(bootloader) slot-count:2</span></span>
(bootloader) snapshot-update-status:snapshotted</span></span>
(bootloader) super-partition-name:super</span></span>
(bootloader) system-fingerprint:OnePlus/OnePlus9Pro_EEA/OnePlus9Pro:11/RKQ1.201105.002/2111112053:user/release-keys</span></span>
(bootloader) treble-enabled:true</span></span>
(bootloader) unlocked:yes</span></span>
(bootloader) vendor-fingerprint:OnePlus/OnePlus9Pro_EEA/OnePlus9Pro:11/RKQ1.201105.002/2111112053:user/release-keys</span></span>
(bootloader) version:0.4</span></span>
(bootloader) version-baseband:</span></span>
(bootloader) version-bootloader:unknown</span></span>
(bootloader) version-os:11</span></span>
(bootloader) version-vndk:30</span></span>
Finished. Total time: 1.181s</span></span></code></pre>
Fuck. Again. ... TBC. AGAIN. how many more yaks are left to shave one wonders.</p>
todo:</p>

skip all the guided setup steps</li>
open the microG app</li>
run the self check</li>
magisk root if needed</li>
everything else</li>
</ul>
config</h3>

settings

sound</li>
advanced

screen locking sound - off</li>
touch sounds - off</li>
</ul>
</li>
system</li>
advanced

gestures

power menu

advanced restart - on</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
Lineage+microG+Magisk again - success!</h3>
Let's try downloading the latest and flashing again.</p>
https://download.lineage.microg.org/lemonadep/?sort=time&order=asc</a></p>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span>
$ ll lineage-18.1-2022040*</span></span>
-rw-rw-r-- 1 tim tim 192M Apr 30 20:38 lineage-18.1-20220401-microG-lemonadep-recovery.img</span></span>
-rw-rw-r-- 1 tim tim 1.1G Apr 30 20:39 lineage-18.1-20220405-microG-lemonadep.zip</span></span>
-rw-rw-r-- 1 tim tim 3.9K Apr 30 20:37 lineage-18.1-20220405-microG-lemonadep.zip.prop</span></span>
-rw-rw-r-- 1 tim tim  109 Apr 30 20:37 lineage-18.1-20220405-microG-lemonadep.zip.sha256sum</span></span>
$ sha256sum -c lineage-18.1-20220405-microG-lemonadep.zip.sha256sum</span></span>
lineage-18.1-20220405-microG-lemonadep.zip: OK</span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span>
$ cd update_verifier-master </span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/update_verifier-master</span></span>
$ python update_verifier.py lineageos4microg_pubkey ../lineage-18.1-20220405-microG-lemonadep.zip     </span></span>
verified successfully</span></span></code></pre>
We'll do a swap slot again so that it won't overwrite the currently working lineage install.</p>
Vol-down + power button to enter recovery (from powered off).</p>
Lineage recovery says "Active slot: b" at the top. Let's boot it, check what we have, swap, boot again to see the other...</p>

slot b: broken install - drops to bootloader</li>
slot a: also broken ...., erm I swear I booted that. wtf. phones.</li>
</ul>
so that means I should swap slot so that "a" is active, so that fastboot flashes over the broken "b" and then swaps to it without touching the currently working "a" slot.</p>
Use lineage bootloader to format the system just to see if that fixes the broken install. .... Nope. Both slots still fubar.</p>
Right, well, flashing time either way.</p>
put phone into recover "update > apply from adb" mode</p>
adb sideload lineage-18.1-20220405-microG-lemonadep.zip</code></p>
long pause...</p>
back > reboot system</p>
hurrah, a lineage boot animation</p>
rattle through setup steps again, did wifi, nothing else</p>
todo: fingerprint and pin</p>
todo: re-root with magisk before doing anything else</p>
new things compared to plain lineage: f-droid & microG apps</p>
installed andOTP & vespucci (OSM editor) from f-droid</p>

enable developer options (system > about phone, tap lots of times)</p>
system > advanced > developer options > usb debugging ON  + Disable adb authorisation timeout (so that if my screen fails or the software misbehaves I can get my files over a usb cable)</p>
Install Magisk by John Wu via F-Droid instead of adb this time</p>
Extract the updated payload:</p>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG</span></span>
$ cd payload </span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ ../../payload-dumper-go/payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go ../lineage-18.1-20220405-microG-lemonadep.zip</span></span>
Please wait while extracting payload.bin from the archive.</span></span>
payload.bin: /tmp/payload_208101762.bin</span></span>
Payload Version: 2</span></span>
Payload Manifest Length: 94295</span></span>
Payload Manifest Signature Length: 267</span></span>
Found partitions:</span></span>
boot (201 MB), dtbo (25 MB), odm (2.9 MB), product (378 MB), system (1.1 GB), system_ext (234 MB), vbmeta (8.2 kB), vbmeta_system (4.1 kB), vendor (1.3 GB), vendor_boot (201 MB)</span></span>
Number of workers: 4</span></span>
odm (2.9 MB)            [==============================================================================================] 100 %</span></span>
boot (201 MB)           [==============================================================================================] 100 %</span></span>
dtbo (25 MB)            [==============================================================================================] 100 %</span></span>
product (378 MB)        [==============================================================================================] 100 %</span></span>
system (1.1 GB)         [==============================================================================================] 100 %</span></span>
system_ext (234 MB)     [==============================================================================================] 100 %</span></span>
vbmeta (8.2 kB)         [==============================================================================================] 100 %</span></span>
vbmeta_system (4.1 kB)  [==============================================================================================] 100 %</span></span>
vendor (1.3 GB)         [==============================================================================================] 100 %</span></span>
vendor_boot (201 MB)    [==============================================================================================] 100 %</span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ ll</span></span>
total 4.0K</span></span>
drwxr-xr-x 2 tim tim 4.0K Apr 30 21:34 extracted_20220430_213414</span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ ll extracted_20220430_213414 </span></span>
total 3.3G</span></span>
-rwxr-xr-x 1 tim tim 192M Apr 30 21:34 boot.img</span></span>
-rwxr-xr-x 1 tim tim  24M Apr 30 21:34 dtbo.img</span></span>
-rwxr-xr-x 1 tim tim 2.8M Apr 30 21:34 odm.img</span></span>
-rwxr-xr-x 1 tim tim 361M Apr 30 21:34 product.img</span></span>
-rwxr-xr-x 1 tim tim 224M Apr 30 21:34 system_ext.img</span></span>
-rwxr-xr-x 1 tim tim 1.1G Apr 30 21:34 system.img</span></span>
-rwxr-xr-x 1 tim tim 8.0K Apr 30 21:34 vbmeta.img</span></span>
-rwxr-xr-x 1 tim tim 4.0K Apr 30 21:34 vbmeta_system.img</span></span>
-rwxr-xr-x 1 tim tim 192M Apr 30 21:34 vendor_boot.img</span></span>
-rwxr-xr-x 1 tim tim 1.3G Apr 30 21:35 vendor.img</span></span></code></pre>
send to phone:</p>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ adb push extracted_20220430_213414/boot.img /sdcard/Download/</span></span>
extracted_20220430_213414/boot.img: 1 file pushed, 0 skipped. 192.7 MB/s (201326592 bytes in 0.996s)</span></span></code></pre>
run the patch in the phone Magisk UI (under “install”)</p>
get the patched file back:</p>
im@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ patched=`adb shell ls /sdcard/Download/magisk_patched*`</span></span>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ adb pull $patched</span></span>
/sdcard/Download/magisk_patched-24300_v4xhs.img: 1 file pulled, 0 skipped. 34.7 MB/s (201326592 bytes in 5.532s)</span></span></code></pre>
turn on advanced reboot again in settings</p>
reboot to fastboot</p>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload</span></span>
$ fastboot flash boot magisk_patched-24300_v4xhs.img </span></span>
Sending 'boot_a' (196608 KB)                       OKAY [  6.597s]</span></span>
Writing 'boot_a'                                   OKAY [  0.611s]</span></span>
Finished. Total time: 7.387s</span></span></code></pre>
"reboot now"</p>
success. open up magisk, shows as installed (v24.3)</p>
F-Droid app list</h2>
Turn On Install history</h3>
Before starting installations do this so that we (might) have an exportable record to share.</p>
Turn on install history before starting so that we can export at the end - https://gitlab.com/fdroid/fdroidclient/-/issues/70#note_89110104</a></p>
Settings > Expert mode > Keep install history</p>
Export list when done</h3>
Settings > Install history and metrics > Share (share icon with 3 linked dots at top) > fail, no apps that know what to do with the file installed yet. gah.</p>
Google contacts, calendar & gmail - fail</h3>
via aurora</p>
Install contacts...</p>

Sign in</li>
Goes to microG</li>
Then to official google sign in</li>
Sign in succeeds</li>
Acc now listed in settings > accounts</li>
google contacts still shows no accounts & no contacts. Damn</li>
tap acc, tap any of the settings (all go to same place) turn on "Allow apps to find contacts"</li>
now google contacts shows the account, but still doesn't sync.</li>
</ul>
Hrmm. Need this or equivalent obviously. Switching will be non trivial.</p>
Same problem for gmail.</p>
Calendar app works and has all events, desktop widget broken (shows no events)</p>
Research</h4>
Alternatives</h5>

https://alternativeto.net/software/google-contacts/</a></li>
https://www.reddit.com/r/LineageOS/comments/7wnwpe/alternative_to_google_calender_or_contacts_sync/</a></li>
https://www.reddit.com/r/LineageOS/comments/8qv3p3/alternative_services_to_google_calendar_and/</a></li>
https://www.reddit.com/r/LineageOS/comments/pt5ce7/are_the_contacts_and_telephone_apps_built_by/</a>

https://f-droid.org/en/packages/com.simplemobiletools.contacts.pro/</a></li>
</ul>
</li>
https://f-droid.org/en/packages/com.simplemobiletools.contacts.pro/</a></li>
</ul>
Making it work</h5>

https://github.com/microg/GmsCore/issues/585</a></li>
https://github.com/tomwiggers/Magisk-ConCal-Sync-MicroG</a>

download Magisk-ConCal-Sync-MicroG repo zip</a></li>
extract, go into inner folder and zip up the four items</li>
copy to phone with syncthing</li>
magisk > modules > install from storage > select the zip > done</li>
reboot phone</li>
boot loop. FUUUUUUUUCK</li>
reported issue for what it's worth, including logcat</code> logs: https://github.com/tomwiggers/Magisk-ConCal-Sync-MicroG/issues/3</a></li>
remove modules

https://www.thecustomdroid.com/uninstall-magisk-modules-in-twrp-recovery-guide/</a></li>
connect phone usb</li>
run adb wait-for-device shell magisk --remove-modules</code> on laptop</li>
boot up phone normally</li>
phone booted into lock screen, phew!</li>
</ul>
</li>
</ul>
</li>
</ul>
Calendar</h4>
bearable, widget broken but app itself works and syncs</p>
have to manually turn on sync for additional calendars</p>
Contacts</h4>
Need alternative, perhaps fastmail</p>

Let's try fastmail's contacts...

google contacts web app</a> > export from google contacts > vCard</li>
fastmail web app > dropdown thing > settings</a> > import > contacts > upload addressbook file > choose the vcf

imported all but two, showed me the errors, not ones I care about.</li>
</ul>
</li>
check fastmail phone app, contacts come flooding in.</li>
but no sign of them in the main phone contacts app. damn.</li>
</ul>
</li>
https://www.reddit.com/r/fastmail/comments/f2akqj/dealing_with_contacts/</a></li>
</ul>
I've messaged fastmail support to ask them why they have a completely separate contact section in the fastmail app rather than using the shared contacts store in Android/LineageOS.</p>
Because I'm short of good options I will reluctantly try the DAVx app that everyone points to. I really don't want yet another sync involved in the whole thing but I also don't want to run and maintain a NextCloud install (which lots of people suggest).</p>

Installing DAVx5

It's listed in Aurora (the play store proxy) and shows "£4.29" instead of the usual "Install".</li>
Pressed money button to see what happens > error "Download failed, app not purchased". Cool.</li>
Go to DAVx5 on play store</a> and make the purchase

concerningly for the future the play store pops an "install to blah device", which is fine now because it knows about my old</em> phone, but what if I didn't have the old phone some time, does that mean I could no longer get paid apps and install them with Aurora?</li>
</ul>
</li>
Back on the new phone Aurora install now works. Win.</li>
</ul>
</li>
Setting up DAVx5

rattled through default wizard without changing (or understanding) anything</li>
interesting warning about oneplus blocking sync that points to https://www.davx5.com/faq/synchronization-is-not-run-as-expected</a></li>
Create app password in fastmail web interface for DAVx</li>
Login to DAVx app with email + app-password ... fail, presumably because it's a custom domain</li>
Use custom base url option - https://www.davx5.com/tested-with/fastmail</a>

Base url: https://carddav.fastmail.com</code> + email + app-password .... success</li>
Create account: account name? (defaults to tim@timwise.co.uk) - "Use your email address as account name because Android will use the account name as ORGANIZER field for events you create. You can't have two accounts with the same name." - probably fine, left default</li>
Contact group method:

"Groups are separate vCards" - default, left as this</li>
"Groups are per-contact categories"</li>
</ul>
</li>
</ul>
</li>
Next</li>
No contacts sync (missing permissions) > Permissions > All of the below to "on" > grant dialogs</li>
Shows two unchecked checkboxes, but don't know what they mean

"Global Address Book"</li>
"personal"</li>
</ul>
</li>
Trying out ticking "personal", tick it and press orange sync now button</li>
woo, main contacts app now full of contacts. sim switch time!!!</li>
</ul>
</li>
</ul>
GMail</h4>
todo, probably set up imap and use k9mail + web interface on laptop</p>
F-Droid app list</h3>

Vespucci (open-streetmap (OSM) editor)</li>
andOTP (one time password generator with backup/restore capability)</li>
Aurora play store proxy - configure as per above section</li>
Syncthing</li>
OAndBackupX - to replace proprietary (but very good) titanium backup

Needs root, provided by Magisk. Without root no apps can access the data of other apps. This is the primary reason that root is so important to me. I want a backup system that I can actually trust because it's not some proprietary cloud magic pixie dust, it's instead a straight-forward grab-the-files-and-put-them-somewhere-safe operation that I can inspect and get to without more proprietary black magic.</li>
</ul>
</li>
KeepassDX</a> - https://github.com/Kunzisoft/KeePassDX</a></li>
k9mail</li>
vlc</li>
markor</li>
antennapod - todo restore state</li>
qr & barcode scanner</li>
aTimeLogger</li>
Editor</a> (replacement for droidedit?)</li>
GPSLogger</a></li>
</ul>
todo - more apps (f-droid where possible)</h3>

VLC</li>
AntennaPod</li>
Audible</li>
Maverick (for recording and viewing gps traces, could do with a replacement, the map tiles seem to have died and it's getting buggy)</li>
google home (for chromecast & google mini integration)</li>
RainAlarm (paid for ad-free)</li>
Google calendar (or find a non-google replacement)</li>
Google contacts (or find a non-google replacement)</li>
Waze</li>
Google maps (for satnav, route planning and finding local businesses)</li>
LinkedIn</li>
Twitter</li>
Telegram FOSS (a slightly free-er version of telegram)</li>
...</li>
</ul>
Other f-droid app suggestions</h3>

https://www.reddit.com/r/fdroid/comments/lzzdbg/a_list_of_some_fdroid_apps_i_installed_to_get_you/</a></li>
</ul>
Play store app list (via Aurora)</h2>
Only things not available in f-droid</p>

maps.me</li>
Spotify</li>
Audible</li>
Firefox</li>
google maps - works flawlessly!</li>
signal messemger

surprisingly not in f-droid!</li>
can't be on multiple phones at once, can only "transfer". Grumble.</li>
</ul>
</li>
banking apps (2/3 work)</li>
sunsama</li>
trello</li>
slack</li>
rain alarm - todo restore paid version</li>
zoom</li>
bulb energy</li>
YNAB</li>
Ring (doorbell)</li>
Google home</li>
Life360 (shared gps locations for family, saves a bunch of "have you left yet?" text messages)</li>
Fastmail - not for the mail, prefer k9mail, but to see if it can do contacts as well as goooooooogle</li>
DroidEdit</li>
Ringo</li>
National rail</li>
Goodreads</li>
Pluralsight</li>
LinkedIn</li>
FreeAgent</li>
BBC

Weather</li>
Sounds</li>
iPlayer</li>
</ul>
</li>
Amazon Shopping</li>
Amazon Prime</li>
YouTube</li>
Microsoft authenticator</li>
Google authenticator</li>
Bandcamp</li>
Offline English Dictionary by Livio</a></li>
Strava</li>
</ul>
Built-in apps</h2>
Camera - configure</h3>

Hamburger

GPS > on</li>
Touch focus duration > infinite (man that has always annoyed me! Default 3 seconds, why?!)</li>
shutter sound > disabled</li>
</ul>
</li>
</ul>
Browser - disable</h3>
Disable (using firefox instead)</p>
Messaging - disable</h3>
Messaging > long press > app info > disable</p>
Individual app setup</h2>
Syncthing</h3>

Open web gui</li>
Disable all non-LAN</li>
Remove default folder (Camera)</li>
Set device name</li>
Connect to existing machine(s).</li>
Add /storage/emulated/0</code> folder (can't do this in non-web gui, stupid security warning. Nanny state of android.)

Warnings about failure to sync Android/data</code> and Android/obb</code>. Looks like they've tightened up security more, either that or syncthing has better warnings now. These folders will be exposed by OAndBackupX so I can safely ignore this until I sort out the ignore file for syncthing.</li>
</ul>
</li>
</ul>
Oh no, syncthing can no longer access /storage/emulated/0/Android/data</code> - https://forum.syncthing.net/t/can-i-sync-send-only-data-from-android-data-in-android-11/17898</a></p>
Even more reason to have root and run a real and independent backup tool.</p>
Syncthing ignore patterns</h4>
Set up ignores for caches and things that won't sync. Files for this are on github at https://github.com/timabell/dotmatrix/tree/master/phone</a></p>

Create file phone.stignore</code> in the root of the sync'd sdcard folder. This one you can edit on the laptop as you find more things you don't want sync'd.</li>
Create file phonetemplate.stignore</code> in the root folder and put only this line in it #include phone.stignore</code></li>
On the phone when they have sync'd copy the phonetemplate.stignore</code> to a file named just .stignore</code> in the root folder.</li>
</ul>
The reason for this dance is that syncthing won't sync the .stignore</code> so to be able to edit the ignores on a different device you have to set up a link.</p>
AndOTP</h3>

Set up password</li>
Turn off global timeout bar (distracting & stressful)</li>
Icons: small</li>
Layout: compact</li>
Copy backup from old phone with syncthing and restore it</li>
</ul>
AntennaPod</h3>
Now I get to the crux of why I was so fucked off with Android.</p>
I want to copy the exact</em> state of antennapod to the new phone; but my old phone de-rooted itself and broke the backups; and you can't back up ~/data where everything lives without root because it's all locked down with per-app-user permissions, which is great for isolating apps from each other ... and you. The only option for those folders is in app sync (e.g.. sign in to trello/firefox/slack etc) so local state is throwaway, or backup to google drive .... to some magical fucking invisible folder so you can't even download the backups it made. This shit makes me so mad.</p>
We already have working systems for backups and security and these fuckers have broken them and replaced them with their own vendor-locked-in proprietary black-box shite that doesn't even work all the time.</p>
Because I'd lost root and usable backups it took me a while to work out how to transfer my latest beloved podcast player state to a new phone. I knew that you could import/export an OPML list of suscriptions but I really wanted to keep the play state of every episode so I caould easily scan the back-catalogues for episodes I'd never listened to dotted amongst the many thousands of episodes I have listned to. Well darm it there is a full export feature I didn't know about. It exports to the virtual sdcard storagea which is kinda like a home directory that you can actually access as a user. I already have the sdcard storage syncing to my other machines with syncthing so getting it from there onwards is a trivial copy-paste on the laptop.</p>
Here's what a quick google</del> ddg</a> showed up:</p>

https://github.com/AntennaPod/AntennaPod/issues/5391</a></li>
https://github.com/AntennaPod/AntennaPod/issues/5031</a></li>
https://github.com/AntennaPod/AntennaPod/issues/377</a></li>
</ul>
Old phone:</p>

hamburger menu top left</li>
settings (cog at bottom) > storage > import/export > database export</li>
ensure syncthing is running and connected</li>
</ul>
Laptop:</p>

Copy file on laptop to new phone's sync folder</li>
</ul>
New phone:</p>

make sure symcthing is running, connected and up to date to pull new file in</li>
hamburger menu top left</li>
settings (cog at bottom) > storage > import/export > database import > Confirm</li>
select file</li>
press ok to restart Antennapod</li>
Woo! Queue entirely intact!</li>
</ul>
Downloads have to be re-done for the queue but the UI makes that dead easy, just hit play and it switches to a download button so you can whip through them quick.</p>
Bulb energy</h3>

App downloaded</li>
Sign in by magic email link failed</li>
Sign in with password worked</li>
Statement download failed with "no apps can perform this action" but can always use website to get them. ... ah, no pdf reading apps installed, installing a random pdf app fixed that. Not a very clear error message that one!</li>
</ul>
KeePassDX</h3>
Enable fingerprint unlock https://github.com/Kunzisoft/KeePassDX/wiki/Advanced-Unlocking#b-link-database-credential-to-advanced-unlock-recognition=</a></p>
NeoBackup</h3>

Backup folder > neo-backup folder on emulated sdcard home > use this folder</li>
Allow access prompt > allow</li>
</ul>
"i" button top right > interesting note about types of android data and inability to restore apps using hardware key stores such as signal</p>


work out what apps to back up -- all of them, why guess what I'll need with so much storage available</p>
</li>

actual backups - done</p>
</li>

schedule backups - done, works great</p>
</li>

maybe automate clearing off phone onto laptop/server</p>
</li>

Set backup password</p>
</li>
</ul>
Set scheduled backup:</p>

7 days</li>
5am</li>
system apps + user apps</li>
apk + obb + data + device protected data</li>
all apps</li>
enabled</li>
Start now</li>
</ul>
Firefox</h3>
3dots > settings > Search > duckduckgo</p>
firefox was active for 6 hours which makes me think it isn't backgrounding properly, perhaps because of media or js on websites.</p>
Firefox icon > long press > App info (i) > Advanced > Battery > Background restriction > Restrict</p>
Signal</h3>
Use transfer account to new phone feature
Enable chat backups to virtual SD: dots > settings > chat. > Chat backups. Set folder and create backup.</p>
Signal popup saying detected issue with play services and establishing own connection. Now have permanent notification.</p>
GPSLogger</h3>
https://f-droid.org/en/packages/com.mendhak.gpslogger/</a></p>

hamburger

logging details

save to folder > give permissions > new folder (+) > ~/gpx</code></li>
New file creation > every time I start</li>
</ul>
</li>
performance

Logging interval > 5 secs</li>
</ul>
</li>
</ul>
</li>
</ul>
So that syncthing can see it. Defaults to inaccessible data folder. Fffffff</p>
Also set to create new track every time.</p>
A thought on moving apps and data between phones</h2>
During the 1000th discussion of phones with my long-sufferring better-half I happened to utter a line that I think realy captures why this whole nonsense is (probably) worth the effort:</p>

"Living in the iOS ecosystem with their magical one-tap transfer from old phones to new phones that leave the new phone exactly in the same state as your old phone ... until you realize one day you want to leave the ecosystem, and you have to leave all your data behind in their walled garden." ~ Me</p>
</blockquote>
There are several ways of moving data between phones, which provide varying levels of protection against data loss disasters. Disasters that could be local (phone down loo), localized (all devices stolen from house), cloud based (cloud account hacked and wiped), a combination (cloud hacked and devices remote-wiped).</p>
Copying files directly</h3>
Something like syncthing.</p>
Only works for data on virtual sdcard such as photos & downloads.</p>
Local backup/restore</h3>
With OAndBackup, NeoBackup, TitaniumBackup etc.</p>
Needs root.</p>
Can get app's private data out to storage you control.</p>
Can also backup set of installed apps (.apk</code>s) which it turns out is actually quite a time saver compared to clicking install in the app store 100+ times.</p>
Vendor-provided cloud backup</h3>
Completely walled-garden solution. No way to get your data off their cloud without restoring to another device they support (i.e. another Android/iOS device).</p>
3rd party cloud backup</h3>
Something like dropbox.</p>
It seems that the duopoly has deemed this unacceptable so have hobbled these apps fom working properly or at all. Dropbox on iOS has been reduced to a glorified photo backup which can't even run in the background.</p>
Personally I think the fact that the duopoly is denying competition is anti-competitive practice the likes of which saw Microsoft get record fines from the EU back in the day. Hopefully the same will happen to Apple & Google to crack this open a bit, but until then we'll have to just shovel digital shit like this blog post to get out from under them.</p>
File transfer</h2>
Easy thanks to syncthing, just move the folders & files on the laptop from sync sdcard folder of the old phone to the one for the new phone. Win.</p>
Trial TitaniumBackup restore - success</h2>
Before my old phone de-rooted itself thanks to a F****g OS update (grrrrr) I did have titanium set up, so I have somem rather stale backups from before then. Mostly as an experiment I've installed Titanium on the new phone to see if the restore works. This is the first time I've had two capable and still working phones at the same time to be able to experiment like this.</p>

Installed TitaniumBackup via Aurora.</li>
Installed Titanium's "key" app via Aurora.</li>
Titanium complained that Play store wasn't responding so couldn't check licence key. Fucksake. More lockin.</li>
Removed key app again.</li>
Copied one of the old backups across (google authenticator). (An app I had not yet installed on the new phone)</li>
Fired up titanium, hit restore for that app.</li>
Boom! There's the app installed, fired it up, worked no problems and now have an (outdated) list of OTPs that I don't use any more. Hollow victory achieved.</li>
</ul>
Clock</h2>

Settings

Time in seconds > on</li>
Alarms

silence after > 5 mins</li>
Snooze length > 15 minutes - I find that if I'm really too tired to move 15 mins makes a good power nap but isn't long enough to be disastrous</li>
gradually increase</li>
vol button to snooze</li>
</ul>
</li>
</ul>
</li>
</ul>
Config</h2>
A complete list</p>
Settings:</p>

Network and Internet

Hotspot and Tethering

Wi-Fi Hotspot

On</li>
AP Band

5 GHz band preferred</li>
</ul>
</li>
</ul>
</li>
Bluetooth tethering > On (to give the laptop internet out and about)</li>
</ul>
</li>
Connected Devices

Connection preferences

Bluetooth

Device name: foo</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
Wifi

Wifi preferences

Notify for public networks > off</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
Apps and Notifications

Default apps

SMS app > Signal (actually a great integrated SMS app)</li>
</ul>
</li>
Notifications

Advanced

Do Not Disturb

People

Calls > "Contacts and repeat callers"</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
Battery

Battery saver and performance

Automatic power saver - 15% (off by default oddly)</li>
</ul>
</li>
</ul>
</li>
Display

Dark theme

on</li>
no schedule - (let's give this a try, we'll see what it's like in broad daylight)</li>
pure black - on (better for battery and I think it looks cooler)</li>
</ul>
</li>
Night Light

Schedule</li>
sunset till sunrise - this is an experiment, we'll see if it is sensible at it</li>
</ul>
</li>
Advanced (because this is extreem l33t haxxor stuff, beware n00bs)

Screen timeout - 5 mins (scary stuff I know, you were warned by the advanced</strong> thing, I hope you know what you're doing. The nanny phone-state will keep you safe, don't you worry your pretty little head about this section. Twunts.)</li>
Styles and wallpapers - yes you read that right folks, wallpapers are an "Advanced" feature. What is wrong with this people?</li>
push a couple of images across to ~/Pictures with syncthing and set them as home & lock screen</li>
Lock screen</li>
Add text on lock screen

Owner Tim Abell <tim@timwise.co.uk></code> (the idea is that people are generally good and this might be the best way to reach me if I've lost it)</li>
</ul>
</li>
Tap to sleep - off (I only ever do this by mistake)</li>
Font size > Largest (I'm gettingn quite long-sighted)</li>
</ul>
</li>
</ul>
</li>
Sound</li>
Vibrate for calls - Always (also has a vibrate then ring which is a nice option)</li>
Advanced (why??????)

Phone ringtone - thriller three (there's a great selection & variety available, impressed with Lineage here. Also liked Solarium, Sheep (yes actual sheep), Rigel)</li>
Default notification sound - Hojus (also liked Beryllium and Titan for minimal stress-inducing feel. Again a great selection available with some really fun sounds).</li>
Default alarm sound - Piezo Alarm (for the retro kicks. Again more winners in here, well worth exploring)</li>
Dial pad tones - off</li>
Screen locking sound - off</li>
Touch sounds - off</li>
</ul>
</li>
Privacy</li>
Show passwords, display characters briefly as you type - off (I just find this annoying/distracting)</li>
Notifications on lock screen

Don't show notifications at all - this is more for a clear mind rather than privacy, though that's a bonus. I hate looking at my (locked) phone for the time and getting drawn into the notifications. Happened toooooo many times.</li>
</ul>
</li>
Trust

SMS message limit - always confirm (default of 30 seems like a lot of spam to me), might go to 5 if this is a pain</li>
</ul>
</li>
Security</li>
Screen lock - pin</li>
Fingerprint</li>
System

Buttons

Additional buttons

Slider top - total silence</li>
</ul>
</li>
</ul>
</li>
Status bar

Network traffic monitor</li>
Display mode > upload and download</li>
Battery status style - Circle (much higher fidelity information)</li>
</ul>
</li>
Advanced

Gestures</li>
Power menu

Advanced restart - on</li>
</ul>
</li>
Developer options

Automatic system updates > off ("Apply updates when device restarts)</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
About phone</li>
Emergency information (wtf is this in about phone for? I had to ddg to find it)

Add information</li>
set all the things</li>
Add contact</li>
doesn't work, just shows empty list to choose from. dang. TODO</li>
</ul>
</li>
</ul>
Legacy phone connectivity</h3>
The old phone has Wifi at home but without a sim has no internet out and about. Tether to new phone to give it access to internet elsewhere as needed.</p>
Connected to WiFi hotspot but although it accepted the details it fails to actually connect.</p>
Pair to other phone with bluetooth (either end is fine). On old phone open bluetooth settings, go to device > configure (cog) > Internet access > On.</p>
Success, old phone has internet via bluetooth to new phone just like laptop. Win.</p>
Todo</h2>

firmware update: https://wiki.lineageos.org/devices/lemonadep/fw_update</a></li>
Backup/restore with titanium</li>
better camera support? https://www.xda-developers.com/google-camera-port-hub/</a></li>
see if android pay etc will work with magisk magic mask pretending we haven't unlocked anything</li>
find an alternative to titanium backup https://forum.xda-developers.com/t/farewell-to-titanium-backup-looking-for-alternative.3932814/</a></li>
contribute to lineage to link to golang extractor, maybe with step by step instructions</li>
Make signal default sms app</li>
Camera from lock screen</li>
transfer paid version of rain alarm, seems to be missing three-dot-menu on new version</li>
Xmeye cctv</li>
Backup alarms</li>
Signal SMS warnings -> Settings-privacy-Trust-SMS Sending Limit > 5</li>
No access to work google cal</li>
Ignore public wifi</li>
Maps.me voice. Needs language pack. Wut.</li>
Remember Bluetooth tethering setting. Needs code change.</li>
Phone default to Bluetooth</li>
</ul>
Problems & bugs</h2>

Phone app

Fails to unblank when phone rings half the time. Workaround to use power button to sleep/wake a couple of times to wake it up. Painful but bearable.</li>
</ul>
</li>
Rainalarm problems:

no menu so can't enable paid copy</li>
map loads but no rain shown</li>
</ul>
</li>
</ul>
OS Updates</h2>
https://wiki.lineageos.org/devices/lemonadep/update</a></p>
Watch out for de-rooting, make sure backups are up to date first.</p>
Got the first Over The Air (OTA) (...well, over wifi anyway) update from LineageOS. Download. Install. Reboot. Magisk no longer installed. Well that's a fucking pain. NeoBackup fails to start. Backups now broken. Fuck you android.</p>
todo: see if I can restore root with out a full wipe and reinstall this time. twunts. what a pile of yaks.</p>

OnePlus Community forum: Keep Root with Magisk with OTA updates</a></li>
piles of fucking yaks courtesy of reddit</a></li>
https://android.stackexchange.com/questions/200410/how-do-i-update-an-ota-while-retaining-twrp-and-magisk-only-with-my-phone</a></li>
https://topjohnwu.github.io/Magisk/ota.html</a> -</li>
</ul>

"When an OTA is available, first go to (Magisk app → Uninstall → Restore Images). Do not reboot or you will have Magisk uninstalled."</p>
</blockquote>
Bit fucking late now.</p>
So basically you have to know all this before</em> you let android install an update and fuck up your setup (again). Pit of failure.</p>
Guess I'll have to re-do the patch thing I did in the first place.</p>
Export Antennapod state, backup andOTP and make sure syncthing has pushed the files before attempting anything.</p>
Re-root attempt - success!</h3>
Let's try repeating the magisk setup from the original install but with the updated image.</p>

Check current version installed

on phone: settings > about > android version >  > lineageos version: lineage-18.1-20220518-microG-lemonadep</li>
</ul>
</li>
Download matching image to laptop from https://download.lineage.microg.org/lemonadep/</a> (if it's still available, which it is. If it wasn't then might have to run more updates or grab image from phone somehow)</li>
Run the payload extaction, patch on phone & reflash as in original install:

 ../../payload-dumper-go/payload-dumper-go_1.2.0_linux_amd64/payload-dumper-go ../lineage-18.1-20220518-microG-lemonadep.zip</code></li>
[plug phone in to laptop with usb]</li>
adb push extracted_20220704_232617/boot.img /sdcard/Download/</code></li>
patch the file in magisk on the phone (install > patch > select file etc.)</li>
get the patched file back on to laptop: patched=``adb shell ls /sdcard/Download/magisk_patched*`` && echo $patched && adb pull $patched</code></li>
advanced reboot on phone > fastboot</li>
flash the boot image:</li>
</ul>
</li>
</ul>
tim@max:~/Downloads/oneplus9pro/lineage-for-microG/payload2</span></span>
$ fastboot flash boot magisk_patched-24300_nspcI.img</span></span>
target reported max download size of 268435456 bytes</span></span>
sending 'boot_b' (196608 KB)...</span></span>
OKAY [  5.675s]</span></span>
writing 'boot_b'...</span></span>
OKAY [  0.644s]</span></span>
finished. total time: 6.320s</span></span></code></pre>

reboot phone ("reboot system now" > power button)</li>
open up magisk app .... drumroll ....</li>
success!! "Installed 24.3 (24300)"</li>
</ul>
... run a backup with neobackup to see if it works... oh never mind it just popped a notification that neobackup has been granted superuser so I guess it can run its backups now and the schedule has kicked it off after a reboot.</p>
OTA (Over The Air) update without losing root - fail</h3>
Having regained root by patching and flashing boot.img again (above) I'm going to try the OTA on-phone patching as per https://topjohnwu.github.io/Magisk/ota.html</a>:</p>

Settings > System > Advanced > Updater</li>
</ul>
Press refresh symbol in top right corner.</p>
Two images available:</p>

LineageOS 19.1, 11 June 2022, 1.3GB > INFO > "Update blocked This update cannot be installed using the updater app. Please read https://wiki.lineageos.org/devices/lemonadep/upgrade</a> for more information."</li>
LineageOS 18.1, 18 May 2022, 1.1GB > INSTALL > ...</li>
</ol>
So let's try the point update following the magisk ota instructions</a></p>

Magisk app > Unistall.... um, where's that then. Perhaps missing because it wasn't instaled this way.

"This will restore partitions modified by Magisk back to stock from backups made at install in order to pass pre-OTA block verifications." - yeah that figures, I didn't do that so there's no backups to restore.</li>
Guess we'll skip this and hope for the best</li>
</ul>
</li>
Settings > System > Advanced > Updater > 18.1 18 May 2022 > INSTALL > OK</li>
... ~ 10 mins elapse ...</li>
Updater shows "reboot" button. DON'T PRESS IT!!</em>.</li>
Magisk app > install > install to inactive slot (after OTA) >

"Attention. Your device will be FORCED to boot to the current inactive slot after a reboot! Only use this option after the OTA is done. Continue?" > OK</li>
Target slot _a ...</li>
failed! dammit.</li>
press little save button top right to save log</li>
</ul>
</li>
</ul>
Failure log: magisk_install_log_2022-07-05T21_12_57.log</code></p>
- Target slot: _a</span></span>
- Target image: /dev/block/sde16</span></span>
- Device platform: arm64-v8a</span></span>
- Installing: 24.3 (24300)</span></span>
- Unpacking boot image</span></span>
Parsing boot image: [/dev/block/sde16]</span></span>
HEADER_VER      [3]</span></span>
KERNEL_SZ       [40823296]</span></span>
RAMDISK_SZ      [10201092]</span></span>
OS_VERSION      [11.0.0]</span></span>
OS_PATCH_LEVEL  [2022-05]</span></span>
PAGESIZE        [4096]</span></span>
CMDLINE         []</span></span>
KERNEL_FMT      [raw]</span></span>
RAMDISK_FMT     [gzip]</span></span>
VBMETA</span></span>
- Checking ramdisk status</span></span>
Loading cpio: [ramdisk.cpio]</span></span>
- Stock boot image detected</span></span>
- Patching ramdisk</span></span>
Loading cpio: [ramdisk.cpio]</span></span>
Add entry [init] (0750)</span></span>
Create directory [overlay.d] (0750)</span></span>
Create directory [overlay.d/sbin] (0750)</span></span>
Add entry [overlay.d/sbin/magisk32.xz] (0644)</span></span>
Add entry [overlay.d/sbin/magisk64.xz] (0644)</span></span>
Patch with flag KEEPVERITY=[true] KEEPFORCEENCRYPT=[true]</span></span>
Loading cpio: [ramdisk.cpio.orig]</span></span>
Backup mismatch entry: [init] -> [.backup/init]</span></span>
Record new entry: [overlay.d] -> [.backup/.rmlist]</span></span>
Record new entry: [overlay.d/sbin] -> [.backup/.rmlist]</span></span>
Record new entry: [overlay.d/sbin/magisk32.xz] -> [.backup/.rmlist]</span></span>
Record new entry: [overlay.d/sbin/magisk64.xz] -> [.backup/.rmlist]</span></span>
Create directory [.backup] (0000)</span></span>
Add entry [.backup/.magisk] (0000)</span></span>
Dump cpio: [ramdisk.cpio]</span></span>
- Repacking boot image</span></span>
Parsing boot image: [/dev/block/sde16]</span></span>
HEADER_VER      [3]</span></span>
KERNEL_SZ       [40823296]</span></span>
RAMDISK_SZ      [10201092]</span></span>
OS_VERSION      [11.0.0]</span></span>
OS_PATCH_LEVEL  [2022-05]</span></span>
PAGESIZE        [4096]</span></span>
CMDLINE         []</span></span>
KERNEL_FMT      [raw]</span></span>
RAMDISK_FMT     [gzip]</span></span>
VBMETA</span></span>
Repack to boot image: [new-boot.img]</span></span>
HEADER_VER      [3]</span></span>
KERNEL_SZ       [40823296]</span></span>
RAMDISK_SZ      [10715780]</span></span>
OS_VERSION      [11.0.0]</span></span>
OS_PATCH_LEVEL  [2022-05]</span></span>
PAGESIZE        [4096]</span></span>
CMDLINE         []</span></span>
- Flashing new boot image</span></span>
- Sepolicy rules dir is /dev/hWU/.magisk/mirror/metadata</span></span>
! Installation failed</span></span>
! Unable to download bootctl</span></span></code></pre>
Guess I'll try the patch of boot again.</p>
Conclusion: inconclusive</h2>
The main thing I've learned from this is the long-standing duopoly of iOS+Android has caused deep and hard to reverse problems in the phone software ecosystem. It's a crying shame really because there is so much opportunity for innovation now that phone hardware is basically done, but instead we get stagnation, pointless features, anti-features and down-right user-hostile behaviour from both we-know-best camps. It really reminds me of the dark years of the browser and operating system wars. Particularly when internet explorer became dominant and website (i.e. app) developers targeted proprietary IE APIs, locking everyone in and nearly killing the competition. The same for windows in its prime (when Balmer shouted developers-developers-developers he knew the apps created platform-vendor lock-in that he so desired).</p>
There are some promising signs: Ubuntu so nearly made a phone, and then didn't hit their funding target (the ubuntu phone operating system lives on in obscurity), The Librem looks like it might be a sustainable Linux based phone & business (Android is not</em> Linux remember).</p>
For me I think I'll do the closest thing I can to dual-boot and VMs which is to have a second burner phone to run prioprietary crap, but have a primary phone that's as open as I can make it, just like I did with Linux and Windows around 2005-2010.</p>
Every person that takes this route is a vote for change. User numbers matter, especially when it comes to app vendors; they won't invest in dead platforms, but they can't and don't ignore growing market share, and if that's free and open it gets supported. Just look at Microsoft rewriting their entire</em> dotnet ecosystem from scratch to be cross platform.</p>
Open source moves slowly, but like an iceberg it's hard to stop when it's claimed the land.</p>
End... for now</h2>
That's as far as I've got so far. I'll be editing this post as I progress with the install so do come and look again. Also suggestions and questions welcome.</p>
I really do hope one day I can de-google my phone just like I de-microsofted my laptop, but today is not that day for me.</p>
Further reading</h2>

https://www.howtogeek.com/358166/using-android-without-google-a-kind-of-guide/</a></li>
</ul>


Being a development team lead
2021-12-10T00:00:00+00:00
Here's my take on what it means to be a dev-lead / tech-lead / lead developer; and why it's time I stepped up.</p>
I'm going to explain what I think makes for a really good high-functioning dev team, and how to get there along with links to essential reading on the subject.</p>
As always contributions and thoughts welcome, drop me a line. If you want me to help with your team just shout. I love to just chat with people about their challenges in digital delivery, whether or not I end up becoming part of the project.</p>
I'm also going to say why I think it's time I shifted my</em> contracting focus from individual contributor to being a lead dev for teams.</p>
How to be an excellent lead dev</h2>
Having worked under many team leaders of all types I have a pretty good idea of what's pleasant and effective. And I'm glad to say that being a nice human being to be around is a big part of being a great leader. Dictatorial, unpleasant and command-and-control leadership are just not the right tool for running complex digital delivery projects</p>
Servant-leader</h3>
The best leaders I've come across are almost deceptively non-leader-like. This is because our culture in the west (hello press, television and film) portrays "leaders" as hard-ass bullies who get the best out of an organisation by ordering people around, knowing all the answers like some kind of god and generally being unpleasant. The reality is that the best leaders don't lead by intimidation, they lead by example, lead by being humble, lead by coaching (see below) and know that relationships and trust are what really matter for teams that can ship and respond to problems.</p>
Culture of trust</h3>
Everyone in the team is doing the best they can with the knowledge and circumstances they have. We can all help each other get better with trust and a good dose of nonviolent communication</a>.</p>
Openness with all</h3>
Make things open: it makes things better</a>.</p>
I've always thought it best that anyone in the organisation who is interested can see every last detail of what we're working on, right down to the code. This does of course require a culture of trust, but if that's missing there are deeper problems to be solved.</p>
There are of course some things that need to be protected such as production keys and the personal data of users.</p>
As an example I ran an end-of-sprint demo for a client with all the major stakeholders present for an important project. The demo consisted of a "red screen of death", which showed that we had been unable to get a particular piece of technology with which the team was unfamiliar to work even after a whole sprint on it. Once the shock of the blunt honesty subsided the senior stakeholders were able to see that something needed to be done to ensure success and helped us recruit an external consultant with expertise in that particular technology. From there on things went much more smoothly. If we hadn't demoed our "failure" then there was a real risk that the senior stakeholders would have mistakenly thinking everything was fine and now action was needed, which would likely have resulted in a complete failure of the project to deliver. Some people seem to like to always paint a positive picture to their seniors (the RAG reports that are always yellow until it's too late to fix anything and they finally all go red, as they people go and get new jobs elsewhere...).</p>
Setting the standards</h3>
Nobody wants bugs, unmaintainable spaghetti code, regressions due to missing test coverage. But how do you achieve this in a collaborative positive way? Just demanding it be so is ineffective and builds resentment.</p>
No team is perfect. No software is perfect. There is only "better", "worse" and different tradeoffs between different pressures on system design and human behaviour. So we should strive for the best, bring the team with us, and accept that we are all flawed human beings with an inability to simultaneously keep in our heads every constraint and piece of the system. In fact software is largely written and architected the way it is because</em> even the best human brains have limited capacity for knowledge of a system. Managing complexity and coping gracefully with humans being human is an important part of the puzzle.</p>
I see the push towards today and tomorrow's "best practice" more as giving permission to the team to strive for it than enforcing. Most developers I know want to be and do the absolute best they can. When team output doesn't meet some criteria (such as test coverage) these are some possible reasons that I'd look at. I'd use the ordering below so that we can assume good faith until proven wrong. I definitely wouldn't start with confrontation or accusation. Assuming the best does not mean giving up powers of last resort should all other options be exhausted, but that would be the last option. Filtering for great people on the way in is much better than undoing poor selection once the team is up and running.</p>

What pressures are being placed on a contributor to behave a particular way, where is that coming from? E.g. is a delivery manager pushing for raw speed and that being interpreted as a need to skip test coverage and just ship bare production code. In that case perhaps some conversations about the tradeoffs being invisibly made are all that it would take, and some defense of the fundamentals that drive the need for "best practice" in the first place (e.g. regression-proofing to allow continuous delivery, and confidence in refactoring to support agility and ongoing quality, see also XP</a>)</li>
Is there a lack of knowledge as to why this good practice is beneficial to the project? If so some mentoring and coaching might be in order. Maybe as part of some pair programming exercises.</li>
Provide formal training (perhaps Pluralsight courses) to fill in the knowledge gaps. Particularly for permanent employees. For contractors perhaps a suggestion of courses they could take to level-up.</li>
Communicating to those involved in line management / supplier management the concerns with not meeting the standards the rest of the team are able to achieve. Discussing additional ways of providing support to enable the individual to level-up.</li>
Finally and this is not a preferred option due to the disruption it causes make the call to formally put someone on a formal improvement plan or move them on.</li>
</ol>
The coaching ladder</h4>
Inspired by Developing Leadership podcast episode 19 about a strong engineering leadership culture</a>.</p>
Different people, cultures and neurological wiring need different ways of setting out expectations. A two-pronged approach of gentle coaching + clear expectation & consequence setting seems to cover all bases.</p>

Listening mode - make sure people are heard in groups and one-on-one</li>
Gently coach expectations - "have you thought about doing x?" / "I observe that your PR descriptions average 10 words" / "here's how writing clear PR descriptions helps your valuable ideas be recognised by the team"</li>
Set out consequences of not meeting expectations - "We require excellent communication in pull requests to the standard of [example]" / "this standard is a requirement of continuing to be a senior engineer here"</li>
</ol>
One-minute manager</h4>
Where course correction is needed (when operating more as a line-manager) The (new) One Minute Manager</a> offers some great advice on keeping it simple and clear for all involved. This suggests that one-to-one interactions take the form of:</p>

One-minute on goals</li>
One-minute on praise</li>
One-minute on redirection</li>
</ol>
If doing only three minutes of this means they can be done more often and more reliably with the whole team this is a win, as often the opportunity for this kind of discussion is lacking due to the pressure to ship. It's important to note that this doesn't mean the relationship has reverted to command and control, only that clear expectations are important for everyone's success and happiness.</p>
Current "Best Practice" and Lofty Goals</h3>
Things we should strive towards doing/having.</p>

Great test coverage

100% unit test coverage (in meaning not just in line count)

Why? Because this enables refactoring with confidence, reduces the chance of regressions and encourages good design (e.g. decoupling, SOLID etc.)</li>
</ul>
</li>
Use TDD (Test-Driven Development)... if it's the right tool for the job. TDD is a useful technique to be sure. It's harder to use when you can't picture the shape of what you are going to create in your head in advance which I think is probably its biggest weakness. The ultimate objective is that it should be impossible to change the intended behaviour of a production code without a test failing. TDD helps with this but isn't the only way. Another way is to alter the behaviour temporarily (locally) after the code is written and make sure the tests do actually fail. So long as the output coverage is good then I don't personally think that being religious about TDD is especially helpful. I think it's a technique that all developers should learn and practice, whether or not they use it for every piece of code they write.</li>
Smoke tests

Some kind of end-to-end "did we break anything critical" that runs on real infrastructure. This usually requires dedicated effort to create beyond the tests created during individual feature work. I recommend putting this in the backlog and ensuring it is scheduled as it's too big for a hero-developer to "just do" unless you've completely lost control.</li>
Visual regression testing</a> - in anything with a UI this can catch regressions and unintentional changes that would be hard to catch.</li>
</ul>
</li>
"BDD" (Behaviour Driven Development) aka specification by example</a></li>
A tool for a job. Use it where it's useful such as allowing business analysts to create and validate example scenarios for complex calculations or flows. Beware of the overhead of example wiring code.</li>
Often confused with UI tests due to coincidental timing of creation with browser-automation. BDD is not the same as UI testing, mkay?</li>
The point of BDD is more that it's a user/business need focussed description of what the software is supposed to do that's then made executable through a translation layer. Do not forget this.</li>
You can get a lot of the benefit of BDD by just writing test method names in the BDD style</a>, such as def the_order_total_should_be(total_price)</code> and then chaining them together in a top level "scenario" method just like you would with gherkin but without all the example parsing overhead.</li>
</ul>
</li>
Continuous delivery (and confidence to ship any time without even looking)

Lights-Out delivery</a> - i.e. never having to manually interact with production servers. Related to configuration-as-code etc.</li>
Use feature flags (related to the true meaning of the oft misused phrase "continuous integration") to avoid long-lived feature branches for things you can't release to the users yet.</li>
</ul>
</li>
Use the best type of project management (yes I used the dirty words!) for your</em> project.

Agility:

Methodologies are fine, but the point is to be able to be responsive to changing knowledge and business requirements. Never forget that.</li>
eXtreme Programming</a> - an important piece of the landscape, with a focus on technical approaches that allow agility (always shippable, fully tested, etc.)</li>
Scrum</a> - a good basis for all projects, especially greenfield development</li>
Kanban - better for maintenance mode projects

It's very important to understand "Work In Progress (WIP) limits" regardless of whether you use kanban or not. This video explains wip limits</a> in a way that is easily understandable. The mathematically truth of wip limits will always be counter-intuitive to my monkey-brain I think. In short, throughput matters more than utilisation of each person/resource.</li>
</ul>
</li>
Lean</li>
Flow</a></li>
Scaled Agile (SAFe)</a> for multi-team agile</li>
ShapeUp from basecamp is inspirational. It's a refreshing change from scrum. I wrote a summary of ShapeUp</a> which, in my humble opinion, is worth a read.</li>
Waterfall is bad. Period. Gantt charts are to be treated with suspicion, along with anyone and anything else that promises complete predictability of software development.</li>
Self-directed teams that own their results and can use appropriate methods for their particular challenges are far more effective than standardization.</li>
Personally I like some kind of blend of scrum and kanban, and definitely no story pointing.</li>
</ul>
</li>
</ul>
</li>
Lightweight agile governance:

Stakeholders showing up to sprint demos (show-n-tell) is much more time-efficient and more powerful than written reports.</li>
</ul>
</li>
Ability to rebuild entire estate from source-control:

Binaries</li>
Infrastructure (configuration-as-code)</li>
</ul>
</li>
DevOps in practice not just in name (the development team owning</strong> production)

This implies a diversity of skills within the "development" team.</li>
Assistance from people with expertise outside the team is of course welcome, but the important thing is the team continues to own their own outcomes.</li>
</ul>
</li>
Some (but not complete) harmonization of technology choices:

Largely consistent technology makes hiring and moving around projects easier</li>
Completely consistent technology prevents using the best tool for the job</li>
</ul>
</li>
Following 12 factor principles</a>.

Pushing beyond the original 12 factor principles - Small Batches podcast episode 7</a></li>
Continuous improvement (of the team, the processes and the system being built)</li>
</ul>
</li>
Pair/mob programming.

Collaborating on just about everything ensures no-one is a single point of knowledge for any one thing.</li>
The sharing of knowledge, skills and techniques improves everyone's skills as we go.</li>
Many problems and bugs are caught even before git add</code> has been run. The later a problem is found the more it costs to rectify it.</li>
</ul>
</li>
</ul>
Estimates</h3>
Story points, t-shirt sizes, predictions, time-based estimates and no-estimates.</em></p>
This is a topic that always causes heated debate with lots of dogma on all sides. Experiences of day-long point-guessing sessions leave developers who just want to ship nervous of anyone who mentions the word estimate or point. Project owners/managers get nervous when they have no idea how much they are about to spend. Deadlines can be real or made up. Cost/benefit decisions have to be made. So what should we do?</p>
I say it depends. And more usefully, only create estimates if you can prove you need them</strong>, and then use the most lightweight way you can come up with of getting just enough information to make a call and go back to cutting code.</p>
Creating estimates has a cost, make sure you know you need it otherwise don't pay the price. Kanban calls this "waste</a>" (i.e. anything that doesn't end up in the final product, which definitely includes estimation work).</p>
Avoid story points</a> - they were basically invented as a way to prevent people mistaking estimates</strong> for commitments</strong> but have taken on a horrifying life of their own. I have never</em> seen them be particularly useful in any of the many projects I've been on, and I have regularly seen them be a big cause of pain, wasted time and discontentment.</p>
Genuine reasons for needing some kind of estimate:</p>

Big-feature prioritization - use t-shirt sizes (small, medium, large, elephant)</li>
Can't miss deadline (I arrived on a project when the release date of a non-existent thing had already been announced by the managing director to all the customers). Do as few of the following as you can: (listed in increasing order of cost and decreasing order of preference.)

A "finger in the air" hunch for viability of the project from an experienced engineer</li>
Consider just doing the minimum viable product (MVP) first and then adding nice to haves once that's done (iterate). Focus on always-shippable.</li>
If you are concerned even the smallest viable thing might not be doable then you might have to create some architecture designs and prototypes up-front to reduce risk of discovering complications later.</li>
If time is tight and even the MVP is looking uncertain then you might have to create actual estimates for more detailed tasks to give you a better idea of the options ahead of you. I have actually done this on a fixed-deadline project. It's time consuming and dangerously waterfall but it might be your only choice. The more granular the task list the more accurate the estimate (it helps compensate for human bias), but on the flip side the more detail you create without building the more likely you are to be wrong about the design and tasks and for it all to be a waste.</li>
</ul>
</li>
</ul>
Planning releases - do you really</em> need to do this give the cost? Consider using feature-flags instead to hold back features if you need to time their release.</p>
If you need to forecast based on points use monte-carlo story-point forecasting as implemented by Chris Bimson</a>. While I like this approach I'd challenge the need for the forecasts in the first place before committing to the significant extra overhead of calculating and tracking points and then turning that into forecast.</p>
Staying sharp</h3>
Even though I accidentally led a couple of teams early in my contracting career I didn't at that point fancy moving "up" the ladder. I had seen people become "dev team managers" and rapidly lose all their technical skills through a combination of atrophy and the never-ending churn in state-of-the-art technology and practice. I certainly didn't want to end up us a mediocre manager with mediocre and outdated tech skills. So I put my head down and kept on coding as an "individual contributor" (sometimes referred to as an I.C.). Since then I've learned and seen some tricks that I think can allow people to stay sharp no matter how high they climb.</p>
The pendulum</h4>
By alternating roles you take on between individual contributor (coder) and team lead roles you can keep the experience of coal-face high-pace shipping fresh whilst also giving more value by helping whole teams succeed.</p>
https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/</a></p>
The non-critical-path task</h4>
I tried picking up a story from the sprint when I was leading a team once. It didn't go well but I didn't know any better. Because my primary goal was team enablement and my time was interrupt-driven I'd end up with the team waiting for my piece to get done. Not a great help to the team's delivery speed. I also found with the team being full time on code and me less than full time I was always just that little bit behind on current thinking in the heart of the codebase, and you don't have to be much out of step to be much less productive or cause more problems than you solve.</p>
Since then I've seen two great approaches implemented by others:</p>

Take on a non-critical but useful coding piece that won't block the team. I watched an amazing product owner (who can also code) built an entire support console microservice while us devs built the main production app. This gave huge benefit of allowing the support folks and BAs to self-serve on things we didn't have time to build to production grade. It kept him close to the actual product. It also kept his Ruby on Rails skills current and sharp. Ace. I shall be copying this trick!</li>
Work on enablement pieces. Make sure CI pipelines are flowing well. Do version upgrades. Tackle bugs that haven't made the team's sprint. I've seen a team lead do this effectively and would recommend.</li>
</ol>
Interviewing / filtering</h3>
Not everyone gets this luxury but being allowed to control who works in your team will do more for your ability to succeed than anything else.</p>
For a phone screen I wrote up my process here: Technical phone screen interview questions</a>.</p>
Beyond a quick phone/video screen my favourite approach is a pair-programming exercise as this is such a rich source of information and matches how we actually get things done.</p>
Coaching & mentoring</h3>
I can't say it better than The Coaching Habit by Michael Bungay Stanier</a>. The follow-up book The Advice Trap</a> is also excellent.</p>
I've often been guilty of unloading all my "knowledge" on others before they've even asked. Now I see the error of my ways and instead as per the book look to nudge and help at people's own pace. Being there to assist and give confidence as needed; leading people to the best answer through questions instead of answers (and being happy to be surprised that it's a better answer than I had in mind).</p>
Decision making</h3>
It's good to have single points of responsibility such as a tech lead or project owner, however that doesn't mean that it's top down control.</p>
As for how to make the best decisions in chaos and complexity, or when thinking seems to have narrowed to group-think this is an excellent guide: Decisive: How to Make Better Choices in Life and Work by Chip & Dan Heath</a></p>
Things to read / watch / listen too</h3>

Blog posts

Maker's Schedule, Manager's Schedule by Paul Graham</a></li>
Being Glue post/talk by Tanya Reilly</a></li>
Command and Conquer and the Herd of Coconuts by Joel Spolsky</a></li>
Normalization of deviance by Dan Luu</a></li>
</ul>
</li>
Tech leadership books

Notes to a Software Team Leader: Growing Self Organizing Teams by Roy Osherove</a></li>
</ul>
</li>
General leadership & communication books

The Advice Trap: Be Humble, Stay Curious & Change the Way You Lead Forever by Michael Bungay Stanier</a></li>
5 Dysfunctions of a Team</a></li>
The First 90 Days</a></li>
</ul>
</li>
Personal development (you'll need this for you and your team!)

Daring Greatly by Brené Brown</a></li>
Nonviolent Communication; Marshall Rosenberg</a></li>
Bonds That Make Us Free; C. Terry Warner</a></li>
48 Laws of Power; Robert Greene</a></li>
</ul>
</li>
Lean / Flow / Kanban agile etc

The Goal: A Process of Ongoing Improvement Kindle Edition
by Eliyahu M. Goldratt</a> (Audible version has sound effects and everything!)</li>
WIP: why limiting work in progress makes sense (Kanban) (YouTube)</a></li>
</ul>
</li>
</ul>
(Personally I like Audible books, I think it sinks in better being told the stories by another human voice)</p>
Who to learn from</h3>

The legend of Joel Spolsky</a> - I've been reading his articles since I first started coding. Remember "the joel test</a>"?</li>
</ul>
Other ways of moving "up"</h2>
These days thankfully moving into management isn't the only way to get a pay raise in many organisations. Places have recognised that getting ever deeper into technology is in itself valuable, spawning titles such as "principle engineer" that carry the weight and renumeration to match the value and experience of a coder who has seen many things and developed many other skills beyond pure coding.</p>
There's also the "architect" route where you can be an expert in deciding what to use and how to put it together from the seemingly infinite choice of technical solutions, patterns, services and infrastructure that will solve pretty much any given problem. There's good money to be made being an architect, if nothing else than because your power to royally screw everything up for everyone that comes after you is probably about as high as it gets in this role! (Yes I've been on the receiving end of good and bad "architecture").</p>
The road to CTO. You can join a small startup and ride the explosive growth, ending up with many engineers and services under your control. Or you can work your way up relentlessly in one or more large organisations.</p>
Me, a leader?</h2>
Why me, why now, why this?</h3>
I'd like to build a product, but I can't just stop contracting and hope for the best. As such I've spent some time considering which direction I should take my contractor life.</p>
When considering changes of career direction I highly recommend the book "Working Identity" by Herminia Ibarra</a> which lays out a framework for thinking about the journey of the change, combined with lots of research and stories of others who've faced the chasm (often but not exclusively in the traditional mid-life point).</p>
Here's some ideas that popped up ("identities to try on" in the terminology of the book):</p>

Systems-integration agency/consultancy</strong> - I had a go at this but seems to be too big a jump for me right now, on the plus side it's kicked off another period of learning and self-development for me and is the reason I hired a business coach</a> which I am eternally grateful for.</li>
Become a "frontend" engineer</strong> - Traditionally I've been a "full stack" or "backend" engineer which fits my engineering strengths of dealing with complexity in code (for example tech debt as discussed here</a>). Way back there wasn't much complexity beyond a bit of jQuery in most frontend things, in fact I didn't come across the frontend/backend specialization for many years. For a long time the designers-who-can-html dominated frontend land. But these days with ES6, Typescript, npm, angular, react, etc. etc. there is now endless complexity on the frontend for someone like me to get my teeth into. With that in mind (and with a view to improving my ability to ship side-projects) I've been updating my languishing frontend skills. (There's not much frontend in my recent work for UK Gov as that's pretty much templated out with the gov.uk design system</a>, and accessibility concerns means that fancy responsive SPAs don't tend to pass muster). Having put the idea out there with recruiters etc. but not getting much interest I think it's probably a jump too far in one go, but either way I'll spend some spare time skilling up.</li>
Lead developer</strong> - I don't really know why I didn't think of this sooner, it's actually a good fit for where I've been and where my life is at now. This is my current identity to try on, wish me luck! The rest of this post is about this.</li>
</ul>
Here's why I like the team lead idea right now:</p>
I think can provide more value as a contract team-lead for dotnet delivery teams. I can still crank code, and still enjoy coding, but helping a whole team succeed is a worthy thing.</p>
The higher rates would make up for it being a less commonly available role (there's always at least two individual contributors for every team lead, and leads are less commonly contractors). And I could always fill in with coder work in any gaps.</p>
Thinking back to my history and current position, I've enjoyed the lead roles I've done, I love enabling those around me, and I've learned some tricks for being able to stay close to the tech without getting in the way (see "staying sharp" above). I can still crank code, but it seems a shame to learn everything I have beyond the code and then not put it into practice.</p>
Experience leading</h3>
One of my earliest contracts was a project where I was brought in as a solo dev with a view to delivering an asp.net project. The first task was to collect the requirements from a sales person (not an end user or user researcher sadly but that's how it was). So armed with my new love of balsamiq mockups</a> I set about turning the explained needs into a set of wireframes, using them as a tool for a tool for discussion and iterating them as I better understood the problem at hand. I then started work building. After some time the CTO sauntered over and casually asked how likely we were to meet the publicised launch date now that we understood what needed to be built. "Zero" I said, as this was quite a sizeable thing on a not particularly long timeline. A lovely chap to work with he didn't pass further comment and disappeared again.</p>
Not much later I found myself with a team to work on the project consisting of a mix of permanent development staff who'd been reassigned to the project. Because I was there first and I was in touch with the people relaying user's needs I was naturally the hub for the team. Additionally I was one of the few who'd practiced agile methods in previous roles. The result was I felt it was far more important that I enabled the rest of my team to be as productive as possible by eliminating things that were blocking them as my top priority. So there I was, being a team lead for a dev team building a new product, making sure everyone was able to get on, talking to stakeholders and running our sprint demos to ensure everyone inside and outside the project team understood how it was going and could play their part. Pleasingly we shipped a usable system in time for the deadline.</p>
Experience mentoring</h3>
I've always helped everyone I can be the best they can, freely sharing knowledge with anyone who's interested. Not just developers either, in my view the more non-technical roles know about the thing we're all working on the more effective they can be.</p>
I officially mentored DfE Digital's first civil-servant developer which was a real pleasure. You can hear a bit more about that on my podcast episode "A retrospective of mentoring, with David Gisbey"</a></p>
Summary</h2>
Well that was a lot more than I expected when I sat down to write, guess there's a lot floating around my head after 21 years thinking, doing and learning.</p>
Wish me luck, and I hope this helps you in some way.</p>
If you want my help with a team just get in touch, I'm always happy to chat.</p>


Technical phone screen interview questions
2021-12-10T00:00:00+00:00
I was tasked with giving an initial assessment of technical competence / fit in a 30 minute phone call.</p>
This tech screen is based on Steve Yegge’s "The Five Essential Phone-Screen Questions"</a>
My lightweight version goes something like this:</p>
1) Coding</h2>
Ask them to describe the code for a console app that prints the ten-times table in a grid. This is intended to get them to describe a nested for loop, or maybe some fancier construct. It doesn’t matter if they nail it, you can tell if they understand how to code from how they talk about it. Bonus for whether they can understand a verbal description (possibly imperfectly described), and throw together a solution in their head.</p>
This isn’t a substitute for an in-person pairing session, but it gives a quick indication whether they have a clue how to code.</p>
It's supposed to be something like this, which is remarkably hard to explain on the phone from my side (no points deducted for my inability to be clear!):</p>
1	2	3	4	5	6	7	8	9	10</span></span>
2	4	6	8	10	12	14	16	18	20</span></span>
3	6	9	12	15	18	21	24	27	30</span></span>
4	8	12	16	20	24	28	32	36	40</span></span>
5	10	15	20	25	30	35	40	45	50</span></span>
6	12	18	24	30	36	42	48	54	60</span></span>
7	14	21	28	35	42	49	56	63	70</span></span>
8	16	24	32	40	48	56	64	72	80</span></span>
9	18	27	36	45	54	63	72	81	90</span></span>
10	20	30	40	50	60	70	80	90	100</span></span>
</span></code></pre>
(gist</a>)</p>
2) OO design (and other structures)</h2>
Ask them to describe how they can arrange their code, giving a hint that I’m after object oriented things. I’m looking for a confident off-hand description of inheritance, interfaces etc.</p>
If they don’t fluff that, ask if they know non-OO approaches (composition, modules, mixins etc) that are used in languages other than C# to see how broad and deep their knowledge is.</p>
3) Scripting & Regex</h2>
Context: It’s not uncommon for microsoft devs to never look outside the microsoft sandbox. This can result in someone spending week(s) writing a program to solve a problem that can be done with one linux command. Times are changing and Microsoft loves linux, but these narrow thinkers are best avoided. This question is almost pointless for ruby devs who live in unix land.</p>
Question: you have a server with thousands of plain-text log files scattered across hundreds of folders and you need to find all the lines with a particular user-id in them to solve a production problem.</p>
Good answer: grep</code> (this means they know of unix tools, maybe probe how wide that is)
Bad answer: I’d start a new project in visual studio and use FileInfo classes…</p>
Follow up question: what if you needed to find an identifier that matched a pattern but wasn’t always the same? (i.e. do they know about regex)</p>
Great answer: I’d use egrep</code> with a regex (or similar tools); goes on to explain a bunch of great tools I hadn’t heard of before</p>
If they have other smart ways (e.g. they know about log aggregation tools like Kibana), guide them back to a limited toolset (“it’s a legacy system, you haven’t got any of that...”)</p>
4) Data Structures</h2>
Do they know the difference between dictionaries and lists, particularly when it comes to speed of lookup.</p>
Related: Do they know about algorithmic complexity</a> - this sorts the self-taughts from the computer-science degree crowd, but if you don’t know this stuff it’ll bite you sooner or later.</p>
See if they are aware of the available structures.
See how clearly they can explain why you’d choose one over the other for a particular piece of code.
It doesn’t matter if they can recall the specifics, it just matters that they’d think to go and look up the specifics when they needed to decide.</p>
5) Bits and bytes</h2>
Do they know why you don’t store money in floating point variables?</p>
Great: use decimal instead because floating point cannot accurately represent decimal fractions and will give you rounding errors that upset the accountants and customers
Okay: always use decimals, but don’t really know why
Bad: I’d use a float for money</p>

So that’s it for Steve’s list.</p>
I also like to ask them how they feel about the following:</p>
6) Testing</h2>
Are they militant about TDD? What do they think about BDD? Are they more pragmatic?</p>
How do they think about why you write tests.</p>
This isn’t likely to be make or break, but might indicate better or worse fit with a team and is good to know.</p>
7) Agile</h2>
What kind of agile structures have they worked in. What did they think about it.</p>
I’m looking for enthusiasm for the way the best agile teams operate, it doesn’t matter too much if it’s SCRUM or something else. Do they seem like they’d be adaptable? Would they push for better?</p>
Have they worked on GDS style teams? (For gov work)</p>
Assessment</h2>
During the debrief I usually discuss at least some of the following areas, focussing on what seems to be the make or break points:</p>

skill / knowledge level: architect & team lead > senior engineer > narrow senior engineer > competent > hand-holding needed > complete liability</li>
communication style, ability to discuss piece of theoretical code</li>
attitude towards tech (e.g. do they demand testing or are they more “pragmatic”)</li>
attitude towards and organisational structures and approaches (i.e. scrum or GDS team structures)</li>
how they would likely fit in with the intended team</li>
</ul>
Some example outcomes of the technical assessment might be:</p>

Strong - knew everything, broad knowledge, could clearly do the job</li>
Qualified positive - can probably build something within some boundaries, but you’d want a strong tech lead in charge to keep things on track</li>
Weak - lacking knowledge of important areas of tech, failed to explain a good solution to a simple nested loop problem</li>
</ul>
Bias and scoring</h2>
To minimize risk of bias sneaking in and giving me a "favourite" based on how similar they are to me rather than how well they did I like to do a written report scoring each area above.</p>


Integrating ShipStation with QuickBooks Desktop
2021-09-27T00:00:00+00:00
A possible customer of my new eCommerce Integration
business</a> popped up with an interesting
problem: how to have orders that are entered into QuickBooks
Desktop</a> automatically create matching
entries in ShipStation</a>.</p>

"How can I get my orders from QuickBooks into ShipStation without my staff manually retyping them?"</p>
</blockquote>
What is ShipStation?</h2>
It is a one-stop order shipping interface for shipping your physical outbound
orders that can in turn talk to all the actual couriers so you don't need to
deal with them individually. It connects to Royal Mail, Hermes, DPD etc.</p>
Connecting ShipStation to QuickBooks Desktop</h2>
There isn't a direct way of connecting them together, so that means we need
something to do the work of pulling orders out of QuickBooks and pushing them
into ShipStation.</p>
Here's two pieces of software that can connect the two systems together:</p>
Connex</h2>

https://www.shipstation.com/partners/connex-quickbooks/</a></li>
https://www.syncwithconnex.com/</a></li>
Connex is a bit like Codeless BPA in that it enables a flow of orders, but specific to QuickBooks rather than having many connectors on both sides.</li>
This page lists exactly what pieces of data will be transferred: https://help.syncwithconnex.com/hc/getting-started-with-shipstation</a></li>
ShipStation help page on Connex for QuickBooks</a></li>
The base price for Connex is $199/month</a> at time of writing.</li>
</ul>
Codeless BPA</h2>

https://www.codelessplatforms.com/business-process-automation-platform/</a></li>
Codeless has a ShipStation connector</a></li>
BPA Platform starts from £175/month billed annually.</li>
We'd need to do some investigation of the method of integrating QuickBooks Desktop with Codeless BPA.</li>
</ul>
Connex versus Codeless</h2>
Both Codeless BPA and Connex have similar monthly costs, and both will need
setting up, so choosing which to go with is going to come down to which option
fits your businesses needs better now and in the longer term. Connex will solve
the immediate problem, but Codeless has more connections you can use in the
future.</p>
DBSync</h2>

https://www.mydbsync.com/workflow/shipstation-quickbooks-integration</a></li>
Cloud and On-Premise offerings.</li>
"Chat for pricing"</li>
</ul>
Unify by Webgility</h2>

ShipStation integration: https://www.webgility.com/integrations/shipstation</a></li>
QuickBooks desktop integration: https://www.webgility.com/integrations/quickbooks</a></li>
From $39/month billed annually</a> at time of writing.</li>
ShipStation integration help page</a>.</li>
</ul>
Automate.io</h2>

https://automate.io/integration/quickbooks/shipstation</a></li>
From free to $159/month</a></li>
</ul>
TradeGecko (retired)</h2>

This is being retired</a>, but mentioned for completeness.</li>
https://support.tradegecko.com/hc/en-us/articles/201825933-ShipStation-Integration-Setup-Guide</a></li>
</ul>
To integrate now, or to change technology?</h2>
As everything moves to the cloud it's worth considering whether to invest
further in your existing desktop-based technology choices, or to take the
opportunity to move to cloud based accounting and ERP systems before looking to
connect them together.</p>
Get in touch</h2>
Did I miss an option? Let me know!</p>
Can I help you or someone you know with something? Reach out!</p>


The death of custom software development
2021-08-18T00:00:00+00:00
I've been around this industry for a long time and have been quietly observing the bigger trends while I fiddle around with the details of software.
Here's what I've noticed: writing software by hand is doomed. Expect that it isn't, and it's a bit more nuanced than that.</p>
I think the industrial revolution and the creation and growth of engineering as a profession is a good way to think about the progression of the software profession in the large.</p>
The long view - the birth of payroll software</h2>
A few years ago I decided to get a bit stronger (being your typically feeble software engineer at the time) and joined a Nuffield gym and took them up on their personal training options so that I wouldn't injure myself picking up the first 1kg weight I found in the gym. The PT I got happened to be a lovely older chap named Bryan, who is a very calm and experienced trainer who really helped me build some useful real-world strength through the medium of increasingly heavy kettle-bells and other such devices, all without injury. Now obviously we got talking as we spent many hours on turning me into a proper human, and it turns out that Bryan used to work for the might HP (the computer company not the sauce) of all things, and was part of the software world for many years before I joined it amidst the dot-com mania. I have heard talk of people writing custom payroll systems in the past, but Bryan actually lived it and it was great hearing the stories. In many ways nothing much has changed; software engineers write custom software to solve some pressing business problem for the big businesses that can afford it, and everyone else gets by with bits of paper or whatever they can pull together. The software engineers complain about the hardware engineers, the hardware engineers complain about the software engineers, the sales people sell things that are expensive to do and the software engineers grumble and try and make good. There are fascinating stories of obscure bugs where you have to drill all the way down to the compiler or hardware to find the true source; the stuff of coder legends. All this was true then and is true now.</p>
"That's all very well", you say, but what's that got to do with me? Well I think some of the systems Bryan worked on are illustrative of the geological shifts in software over the years, and are indicative of changes happening to this day and still to come.</p>
In that era, there was no good off-the-shelf payroll software that could solve every business's payroll needs, and every business needs to pay its staff. Doing it by hand is laborious and error prone which makes it a good candidate for turning into software. The in-house custom coding crowd got there first and made it work for big businesses that could afford development teams or the equivalent consultancy fees. Then vendors started to show up but there were edges that they couldn't handle so there was still room for custom code, and now it would be [almost] madness to start a payroll software company (though you might find a way to disrupt the existing players like Sage, but it's not going to be all green fields for the taking like it was back then).</p>
The trend of commoditization and the shrinking need for custom code</h2>
"So what", you say, "there's still vast plains of untouched lands for software development, coding will never die! Coders unite!"</p>
As the really big universal problems are picked off one by one there is indeed still plenty of room, for now at least, for custom code.</p>
(By big universal problems I mean things like: accounting, payroll, Enterprise Resource Planning aka ERP, development tooling, website creation, online payments, social media, buying and selling privately online, car-sharing, short-term property rentals, travel bookings, etc. etc.)</p>
Some organisations will always think they are too special or have too much ego invested in rolling their own (not-invented-here syndrome) to buy an off the shelf options. This is inefficient, but will happen nonetheless if they want to pay for it.</p>
There will always be innovation and disruption, which will require engineering effort. Even in the age-old physical realm of engineering such as building roads, bridges and buildings there continues to be innovation and a need for skilled highway and bridge engineers.</p>
Undeniably, however, there are whole classes of problems that software is capable of solving that have already been solved (some better than others of course), and a market of off the shelf or open source software dominates each particular problem leaving little room for expensive one-off custom builds to solve the same problem again.</p>
As each vendor gets better at solving the problem at hand (or gets disrupted when they fail to), more and more use cases that used to require custom code are served by off-the-shelf software eliminating the need for developer time and replacing it with more-or-less need for installation, setup and configuration expertise.</p>
The end-game for software development</h2>
In the very long run (maybe another 100 years?) I see us getting to a steady state much like exists in traditional physical engineering. There are no vast new uncharted problem spaces waiting to return billions of dollars just for being first to get there. There will still be the occasional disruption but we'll look back on the dot-com boom in the late '90s just like we look back on the industrial revolution and the creation of railroads for the first time. We'll still have railroads, and we'll still have social media platforms and SaaS businesses; and sometimes one company will die and be replaced by others; but we won't be awestruck by silicon valley's ability to turn the world on it's head any more. It will just be normal.</p>
As for the software developers and their trade; they won't go extinct, they'll probably have the same job titles more or less. There will be more developers working for the SaaS companies, infrastructure companies and boxed software companies (locally installed software will never die), and less developers working for businesses that aren't fundamentally software businesses. The latter will just buy the outputs of the software companies for far less than they would have had to pay a development team or outsourcer.</p>
Developer salaries</h2>
Companies that have money will always pay good money for good people, and software businesses serving worldwide business and consumer markets with software that costs nothing to replicate will always have plenty of money.</p>
The "shortage" of software developers caused by the collapse in interest in computer science degrees after the dot-com bust combined with ever increasing production of software has already started to be mitigated as the "boot camp" model that has been running for a few years now at scale is starting to result in volumes of quality and sufficiently experienced engineers. This will push prices for the "average" developer down. In fact prices of the average developer haven't really moved much in the last 20 years in spite of the effects of inflation eating away at the real value of that income. £50-60k is still a reasonable salary for an experienced engineer in an average job in the UK just like it has been for years, while house prices have rocketed up in that time.</p>
So in the long run apart from people that can make themselves invaluable to the richest companies (which takes more than being able to write Python or C#) will still be well rewarded, but the days of exceptional salaries for anyone that can write a for</code> loop let alone recursion are well behind us now.</p>
The fractures and integrations between software</h2>
An interesting effect of the ever-growing amount of coverage of off-the-shelf software and SaaS companies in the global problem-space of business is the rising challenges and expectations for making it all work together (often known as "systems integration").</p>
For sectors where there are many similar businesses (think plumbers, accountants, travel agents), the software vendors will produce solutions that cover everything that the profession needs all in one offering (marketing, finance tracking, invoicing, customer contact etc.). They can do this cost-effectively by selling the same thing repeatedly for near-zero marginal cost.</p>
Bigger more complicated businesses, or more unique businesses are left trying to avoid building everything they need from scratch and instead trying to cobble together a complete solution for their business from all the various SaaS and on-site offerings that are out there.</p>
As businesses put solutions in place for each problem they have to solve, they then start looking at how well these things are working together, and mostly discover that they aren't. One SaaS thinks it has the answer to everything but is wrong, another on-site system thinks it owns the data about customers but the cloud-based CRM has got out of sync again and they keep upsetting customers by getting their facts wrong. So sometimes they just paper over the cracks with people based processes and spreadsheets. Sometimes they hire developers to try and glue systems together that really don't want to be glued together (through the magic of dubious quality APIs or maybe direct SQL database access).</p>
Some bits of software provide value as a standalone thing (e.g. document or chat tools), but even with those there's a drive to integrate (e.g. single sign-on to avoid access for previous employees, auditing and storage for compliance etc). Other bits of software are very painful to leave disconnected (e.g. a storefront in Magento where customers purchase goods and an inventory ordering/management (ERP) system in SAP Business One that makes sure there's anything to ship).</p>
To solve this latest self-inflicted pain of the software world a whole new class of software has emerged who's only job is to join systems together that don't know how to connect directly, or to act as an orchestrator of all the flows of data. You might have seen IFTTT (If This Then That) which is the consumer-centric version of this where you can make any number of events in one SaaS trigger actions elsewhere. Another is Business Process Automation (BPA) such as Codeless BPA which connect all sorts of enterprise software together by hook or by crook, while giving a central place to manage the flow of data and massage and monitor it on the way through as needed.</p>
But even that</em> extra layer of connection still doesn't always give us the magical point and click connection we were promised decades ago where you can make your whole business work without talking to any nerds. For simpler cases you might get lucky, but as soon as you have to make a web hook work many people just glaze over and reach for the phone. And if something stops working between two bits of software that don't really like each other, and your business's whole income depends on that data flow, you're going to do whatever it takes to make it behave again. So in a slightly different shape we find ourselves yet again in need of a vast army of technical folks who really know what's going on and how to work the magic to make the wheels turn again and the ones and zeros flow like a spring down a mountain-side.</p>
Service buses and event sourcing</h2>
One of my tech friends mentioned this (very technical) approach to solving the pains.  This is a really good point, as you can get better more scalable, more fault tolerant and more responsive systems with these approaches. The move to this approach is however contingent on either the vendors playing ball, or on your systems being sufficiently hand-coded already that you can make these kinds of architectural improvements. YMMV as they say. An interesting aside that's worth being aware of and considering. I won't go into what they are here as there's tons of content out there on this.</p>
AI (Artificial Intelligence) and Machine Learning (ML)</h2>
A hot topic of the day is AI and ML. These are growth areas for good reason as they can eliminate whole swathes of formerly labour intensive work, but personally I don't see programming in any meaningful sense as being one of them. I might be a bit biased of course. If anything this is likely to spawn yet more need for highly capably technical folks to build and integrate these new systems, regardless of where the boundaries of each system sits.</p>
A new consultancy</h2>
Yes there was a reason I was even thinking about this, let alone writing it down, but this isn't really a pitch; I genuinely find this stuff interesting to think and write about.</p>
I'm building a new systems integration business off the back of my experience in tech, so if that's relevant to you drop me an email mailto:tim@timwise.co.uk</a> and let's talk. I'm looking to build my network of people who I/we can help, and also my network of people who might want to be involved in the new business.</p>
A penny for your thoughts</h2>
I'd love to talk to you on an actual phone call</em> if you thought this was interesting or relevant (or wrong or silly!)</p>
Drop me a line on mailto:tim@timwise.co.uk</a> and arrange a call!</p>
Footnotes</h2>
Many names of non-custom software</h3>
Sometimes you may hear software referred to as COTS, which is short for "Commercial Off The Shelf" software, or "boxed software". These days you might call that codeless, no-code or low-code. Or maybe "apps". The point here is that you don't have to pay developers to solve a problem (which is expensive, slow and error prone - good for developers paid for their time and skills, not so good for the business paying the bill).</p>


Why I want you to use slack threads
2021-07-08T00:00:00+00:00
Someone has probably sent you a link to this blog post in response to you posting multiple messages in a slack channel on the same topic.</p>
Don't worry, we're not angry with you, we just want to help. We also know that the slack interface can be a bit confusing, especially on mobile, and it's easy to accidentally respond in channel instead on a thread.</p>
This post is not intended to shame or berate you, just to share information on how we can all make slack a nicer and less distracting thing to use for everyone.</p>
</p>
The above screenshot is an example of two consecutive messages in slack, this happens if you send the message in two parts. (Send one message, think of something extra, send another message.)</p>
This post is here to explain to you with kindness why posting multiple messages like this might be causing issues for other people and offer some alternate approaches.</p>
Why multiple messages are a problem for others</h2>
It seems harmless to just post another couple of messages, right?</p>
Slack in any active organisation can quickly become a firehose of information that can be quite hard to keep up with. It takes some effort from all involved to protect everyone's focus and flow time. As an aside if you want an insight into just how damaging distractions can be I recommend diving into Cal Newport's writings https://www.calnewport.com/books/</a>, particularly "Deep Work". Some of us have the luxury of turning off slack for most of the day, but some people need to keep up more actively than that.</p>
Notifications and unread messages</h3>
Slack allows you to carefully manage what notifications you receive, but for a channel it basically comes down to notifications on or off for channel messages.</p>
If you have a channel that you want (or need) to keep up to date with in a timely manner then when someone sends a new channel message you will have intrusive notifications and as shown below an "unread channel" in bold-and-white sat glaring at you waiting to be read:</p>
</p>
If you use the convenient "All unread" feature then that will also sit glaring at you as below with an eye-catching "1 new message" button until you go and look:</p>
</p>
The problem with multiple related channel messages in this case is that if someone gets your first message, decides that it's not a conversation relevant to them and then goes back to trying to concentrate your second message will set off all their notifications once again and send them back to having unread messages to worry about.</p>
If you had instead used the below alternatives of editing or threads then there would be no new notifications and no new unread messages when you edit your message or add to the thread.</p>
Deciding what conversations to follow</h3>
As someone who is trying to avoid slack-overwhelm the ability to follow and unfollow threads is a very useful feature. Slack will automatically give new notifications for threads where you have been mentioned with an @</code> or have added a message of your own to the thread.</p>
To follow/unfollow a thread, hover over the top message in the thread, click the dots, and click "turn off notifications for replied" as shown below:</p>
</p>
If people are not correctly using threads to group themes of discussion then this feature becomes mostly useless.</p>
As an aside this applies just as much to failure to start a new thread when the discussion changes topic as it does to failing to use threads in the first place.</p>
Catching up with a channel</h3>
Sometimes you will join a channel and then mute it (right-click, mute) because you want to be able to keep up to date on your own schedule, say daily or weekly.</p>
</p>
When people in the channel have conversations in the form of channel messages you can easily end up having to read 1000 messages to just know what's been going on for a day and catch up on anything important.</p>
</p>
If instead the people in the channel are disciplined in using threads then it can easily be down to 20 messages with some very long threads that you can dive into if you feel the need.</p>
What you can do instead</h2>
By using the following methods you can help everyone else in your slack channel make better use of their time, more easily follow relevant conversations, and have fewer distracting FOMO moments (Fear Of Missing Out).</p>
Start a thread</h3>
Start a thread on your own message and add more messages in there:</p>
</p>
This allows you to add further context etc for those interested without writing a massive message in one go directly in the channel or creating a string of independent channel messages.</p>
</p>
See also https://slack.com/intl/en-gb/help/articles/115000769927-Use-threads-to-organise-discussions-</a></p>
Edit your existing message</h3>
Did you just forget to mention something? You can edit the message you've already posted to make corrections or add more information.</p>
Hover over the message and then press the "more actions" dots:</p>
</p>
</p>
See also https://slack.com/intl/en-gb/help/articles/202395258-Edit-or-delete-messages</a></p>
Delete your subsequent messages</h3>
If you've already posted more than one channel message and then realise your mistake, you can delete the extra messages before converting to a thread or edit, leaving the channel nice and tidy for anyone who shows up later to read.</p>
</p>
This is particularly important to avoid ending up with multiple threads on different channel messages (on the same topic) as other people reply to different bits of your message resulting in confusion and a disjointed conversation.</p>
</p>
See also https://slack.com/intl/en-gb/help/articles/202395258-Edit-or-delete-messages</a></p>
Offering gentle reminders with an emoji</h2>
If your team knows to do this but forgets sometimes then add :start_a_thread:</code> as an emoji (under "customize workspace) to easily remind people when they forget.</p>
</p>
(Thread gif image source: https://slackmojis.com/</a>)</p>
</p>
Summary</h2>
So in summary, please give a thought to others before you send a second message. It may be quick for you but if there are 50 people in the slack room then you are costing them much more time and attention than it saves you by not taking the time to construct a clean message or thread.</p>
Further reading</h2>

Equal Experts' slack guide</a></li>
</ul>


GitHub rebase and squash considered harmful
2021-03-15T00:00:00+00:00
Not to be confused with git rebase</code> and the ability to squash commits locally with interactive rebase, which is a whole other flamewar</a>.</p>
</p>
This post is predicated on giving a crap about the git history you and your team produce. If you don't care about good quality git history then you are wrong because git history is an important aspect of your engineering output</a>.</p>
In short, "squash and merge" is never good, "rebase and merge" is almost never good, "create a merge commit" is a good default, ... and if you have a really great team then mainlining while pairing/mobbing can be great for mean-time-to-recovery but shouldn't be the only allowed way.</p>
If you don't believe me yet, then let us now go reaaaally deep on all the details of all the ways you can do this and what makes them so good/bad/evil:</p>
"Rebase/Squash and merge" - both bad</h2>
Personally I dislike the "rebase</strong>" and "squash</strong>" buttons on GitHub because:</p>

You don't find out what you actually put in main</code> until after it's done. If you don't believe me check the sha1 of the tip of your PR branch with the new tip of main</code>, your sha1 is nowhere to be seen. Github then has the gall to sign the commits as you if you've let it.</li>
There is no record of which commit you based your branch on, so if there's a new incompatibility because main</code> has moved on there is no record of what commit your branch was based on to assist with figuring out where the problem was introduced, it just looks like a bad patch to main with a PR that could never have worked.</li>
The generated commits lose the true commit metadata generated locally (authorship, timestamps, sha-1).</li>
GitHub generates brand new commit(s) at the point of applying to main</code> with no chance to review first.</li>
</ul>
These two options behave in ways that are not immediately obvious (to be fair github improved the labels on the options after I first ranted about this, but it doesn't change the fundamental issues). And if you press one of them without realising what it's going to do, tough because it's already in main</code>, and if you have branch protection on (you  probably should) then you can't even undo it; it's there foreeeeeeeever.</p>
Someone reviewing your PR has no idea which of these three buttons you will push, which could have been the difference between approving (based on a tidy merge and nice branch history) and rejecting a PR (based on including junk history or a badly worded squash commit).</p>
Let's dive in to the hell that github have created for us:</p>
"Squash and merge" - pure evil</h3>
This one takes all your carefully crafted commits (or pile of "wip" junk</a> if you are lazy or plan to squash) and combines them into a single commit, pausing to allow you to write a better commit message (which few people I've worked with bother to actually do, immortalizing "* wip" in the history forever) and then pushing it to main</code> without further opportunity to review what you are shipping.</p>
The behaviour this option encourages, nay, mandates, is to create gigantic 100-file patches that make your eyes bleed to read, basically making git blame</code> useless beyond knowing who to hate.</p>
Anyone who tries to create many small meaningful and logically coherent patches but who has to use this option has the faustian bargain of choosing between making their team review piles of small PRs (too much overhead) or generating one stinking commit in main</code> with all carefully crafted iterations lost to oblivion (for example commit 1: refactor everything, commit 2: make important one-line business change, commit to main: both at once, argh).</p>
Note that github alters your original commit message to retro-fit PR numbers. Also evil.</p>
If you make a change to a file that results in both renaming and modifying files such as a rename-refactor, you can make that easier to follow by doing it in two commits... unless you squash them together. This is exacerbated by git's rename detection</a> which is actually not stored at all, but instead done as a heuristic when viewing patches. If you split the file name change and contents change into two patches you can help a future reader follow what you did, but if you change a file too much and then squash it all together then git will treat it as add and delete, breaking the ability to follow a line of code back through history without even more detective work and guessing.</p>
I have nothing good to say about the "Squash and merge" option. If you don't want to see the messy branches of your developers then use --first-parent</code></strong>. This option in github solves a non-problem and just makes everything worse. Anyone who thinks this option has any merit is wrong.</p>
If an individual developer wants to create a single squashed commit for main</code> they should squash locally and PR that</strong>. This option adds precisely nothing to what is possible, encourages reviewing and shipping junk history and encourages bad behaviour and unreadable patches. If you think it annoys me you're right, it should never have been created. It enables shit developers to carry on being shit and generate less "noise" in the history. Why would you want a process that only exists to to take the edge off being around the worst developers?</p>
"Rebase and merge" - 99% awful</h3>
This option takes your commits, rebases them on main</code> and then fast-forwards main to the new top commit.</p>
When this option has been used you can no longer tell by looking at the git history that there was ever a PR, which is actually useful context when looking back to work out why the hell something is how it is. (If you look at the commits in github it does magically show the PR number but you can't find that context any other way.)</p>
This option is marginally less offensive to me if used judiciously by skilled teams. You might sensibly use it because your PR was only for peer review rather than because your commits deserve a logical grouping. If you can trust your team to intelligently decide when the commits in a PR would look nice as straight-line history in main</code> then perhaps leave this option available.</p>
If you want your github workflow fool-proof and "scalable" then disable this option. Noobs will find this button and push it for the wrong reasons and give you a shit git history.</p>
The upside of being clever occasionally is outweighed in my experience by the beautiful consistency of everything coming in as "merge PR NNN for feature YYY" and then being able to either ignore the details of the branch with --first-parent</code> (which you can't do if someone has used this button), or look at the commits in that PR as a logical group on the second-parent side when you need more granularity (easily done by running a first-parent log on the last commit of the branch before it was merged).</p>
"Create a merge commit" aka "github flow" - good</h2>
My preferred workflow is to use the normal "merge" button every time. This is sometimes know as "GitHub Flow</a>".</p>
In my view you should generate your commits, sha-1 and all, exactly as they will be merged into main</code>, and then as a separate merge commit those should be combined into main</code>.</p>
To avoid many "tramlines" do local rebases to avoid PR branches being based on very outdated main</code>.</p>

This retains the commit exactly as you crafted it locally.</li>
It calls out the difference between you writing your commit and deciding it can go into main</code>.</li>
It retains the information about which commit you based your branch on.</li>
</ul>
There's something to be said for being able to do a git log --first-parent origin/main</code> and get a consistent list of merged PR branches. It's really quite readable compared to a mishmash of the different styles of merge to main</code>.</p>
You can configure GitHub to disallow the other options if your team are on-board with the idea.</p>
You could argue that single commit PRs are a lot of overhead for something trivial, but equally having to decide each time results in more time worrying about whether something justifies the special treatment of ending up directly on main (in GitHub's modified form), and getting it wrong sometimes.</p>
If your PR branches are full of "wip", "fix tests" and other junk then the answer is not to throw this option out, the answer is to get better at creating good patch sets</strong> by learning to use git rebase --interactive</code> and thinking harder about what would make good patch before you start typing in your IDE. (And noticing when your patch is getting messy and stopping to reflect on good patch generation).</p>
I'm aware there are counter-arguments, but in my experience on many teams this seems to end up being the cleanest balance of trade-offs. If you are an individual developer on your own project then pure mainline development can make more sense, though even then thinking of feature branches and having them explicitly merged where appropriate can neatly group related commits. The only compelling argument I've heard against this flow is mean-time-to-fix, but that doesn't mean you have to kill this approach entirely, just be able to use other approaches too.</p>
But my interface can't do first-parent!</h2>
Someone once complained at me that we shouldn't do merge commits because they make no sense when viewed in github's deficient history viewer, which shows a random mishmash of mainline and branch commits with no indication of which is which.</p>
Seriously, use a better git viewer, there are literally thousands.</p>
It 100% sucks that github, the flagship platform for git, still to this day does not have a proper branch view let alone first-parent view. Appalling. I'm even more horrified that GitHub desktop</a> has inherited this idiotic denial of branch based views. If the view of history / PRs in github doesn't show you what you need don't use it</strong>. Did you know you can review PRs by just pulling down the branch and looking at it locally? You are a capable programmer who uses many tools not an idiot that needs to be spoonfed powerpoint presentations.</p>
Even visual studio can show you branches, merges and first-parent views, it's just a little tricky to find because they insisted on calling everything by different names and using mystery-meat buttons.</p>
Sorry, but I don't have much sympathy with the "my tool is shit so we should do a worse job" complaint against doing things properly, it's weak.</p>
The case against PRs</h2>
The only team I have ever come across that (incorrectly) disabled the "create a merge commit" option did so (I believe) in the name of "mean-time-to-recovery". This is however a false dichotomy.</p>
If you want fast "mean time to recovery" (MTTR) you need to be able to ship a patch to main</code> and on to production fast. There is nothing about the ability</em> to create merge commits or use PRs that prevents that. So long as you don't enable the "require a PR" option in github, there's nothing to stop a team pushing straight to main</code> when that's the right thing to do.</p>
I like the idea that the requirement is not a pull request, but just two pairs of eyes on everything. That gives you the ability to be more responsive as pairs can now ship straight to production. That said, I just don't buy the idea of "mainline development" for every patch</a>. Some changes are just more complicated than a single patch and it's good to be able to break things down in a branch before merging. Don't get me wrong, I'm all for pulling groundwork out and mainlining it (if your dev team is good enough to do this) so that your final patch is smaller, but sometimes a feature change has to go in in one commit to main</code>, and without a branch plus a merge commit to main it is just not granular enough to make a useful history. And no, feature flags will not fix this for you, they are a useful thing for sure, and can reduce the size of unmerged branches, but sometimes you just have to change fundamental things that can't be "flagged".</p>
Team culture is worth considering, regardless of tool configuration. If your team does everything by PR then when something is on fire they'll probably raise a PR and then sit around waiting for approval; and if your team always commits everything to main as they go they'll recover fast but their history will be full of mis-steps and fixups that could have been dealt with before they ever hit main. As always extremes are bad and anyone who fast-talks you into believing one extreme is a panacea is glossing over important nuance and interesting counter-examples.</p>
Disabling merge commits doesn't prevent people from having slow PR based async processes, and equally, enabling merge commits and PRs doesn't stop you shipping a quick patch by pushing straight to main when something's on fire. The two things are orthogonal.</p>
The Review part of PR</h2>
The existence of a Pull Request implies a review by another developer before merging to main</code>. The series of commits, including their summary, description, author and gpg signature is something that we can review and provide feedback on as part of the review process in order to provide a high quality history for those that need to understand why code is how it is, sometimes years in the future.</p>
I have personally heard many developers say "Don't worry about the crappy history I'm going to squash it". Note the future tense in here. So the reviewer is supposed to just approved the awful "wip" history, or ignore it entirely, and trust that the person that does the merge will pick the right button and generate a sane commit to main.</p>
The branch history in the PR that was reviewed and approved is then thrown away and something completely different goes in to main without any opportunity for the reviewer to object or comment.</p>
What is the point of reviewing something and then committing something completely different to main</code>?</p>
The generated history does have value, should be of good quality, and peer-review is an important part of keeping that quality high.</p>
Perfection?</h2>
So what's the ideal? In my view it's the one of the following depending on the quality of your team and what your goals are:</p>
High performing team with top-end developers - reducing mean-time-to-fix</h3>

Two pairs of eyes rule.</li>
Don't disable anything, just educate and trust

(... apart from branch protection to avoid force-push to main because forward-only is good)</li>
(...... and you can probably tell I wouldn't mind if you did disable squash and perhaps rebase given the above ranting).</li>
</ul>
</li>
Use judgement for when to pair, mainline, PR etc.</li>
Practice all approaches so that team is used to fast fixes.</li>
</ul>
Trust your team to use the right tool in the right moment. Big hairy feature? Break it down, use PRs if useful, ship as a merge commit with nice history and link to story and PR. Production on fire? Pair/mob and mainline that sucker.</p>
Mediocre / mixed team, avoiding breakages over fast-fix</h3>

Require PR for all changes.</li>
Disable everything but "create merge commit".</li>
Review PRs for quality of commit list and require rewriting branch history till it's up to team standards before merging.</li>
</ul>


Link your outlook calendar with your google calendar
2021-03-11T00:00:00+00:00
If you have a Google/GSuite calendar and a Microsoft Teams/Office365/Outlook calendar you can get the Microsoft Outlook calendar to pull in all the events from your Google calendar which is handy when you want to know if you have any gaps or conflicts.</p>
Amazingly if you move google entries around in the outlook calendar they changes are pushed back to the google calendar.</p>
Link the Google calendar into the Outlook calendar</h2>

Go to your Teams calendar</li>
Click the "add calendar" button</li>
</ul>
</a>
Here you can see the "Add calendar" button as well as the end result of showing the two calendars together (google in red, outlook in blue)</p>

Click "Add personal calendars" and follow the instructions. The below shows the result. (The calendar defaults to blue but we can change that).</li>
</ul>
</a></p>
Give the Google calendar a different colour</h2>
You can then give the calendar a different colour in "Edit my calendars"</p>
</a></p>


House buying and selling tip - make contact!
2021-03-04T00:00:00+00:00
I've learned a useful trick as part of buying and selling house(s) that I want to share with you just in case you find it useful. It could save you months of stress and delay, and maybe even save a deal that was going to fail.</p>
The big secret</h2>
Establish direct contact with the buyer/seller.</p>
This will allow you to co-ordinate on making the sale happen and spot anywhere that the chain of communication has broken down. (E.g. both solicitors waiting for each other and a message between them that has vanished into the ether).</p>
But how?</h2>
This should be easy, but estate agents don't want you to go round them, and solicitors won't necessarily pass on your request. But there is a way round this.</p>
Buying a house</h3>
Simple, put a note through the seller's letterbox offering direct contact and including your contact details.</p>
Here's a template note to a seller</a> that worked for me in a previous purchase.</p>
Sample letter to sellers</h4>

To:

The Owners,

[Address of house

you want to buy]</p>
[Your name here]

[Your address (optional)]</p>
Contact: [01184960000 / your.email@example.org]</p>
[Date]</p>
Hi,</p>
We’re the people who put the offer in on your house, and we just wanted to give you the opportunity to make direct contact should you have any questions.</p>
We still absolutely love your house, and are very keen to proceed with the purchase. We are working with [Estate Agent] to move the sale of our house forward.</p>
Our contact details are above, don’t hesitate to get in touch should you wish to. We will of course still do all formal communications through the estate agents and solicitors, we’ve just found it handy in the past to also have direct contact.</p>
Yours,</p>
[Your name here]</p>
</blockquote>
Selling a house</h3>
Write a formal letter with an offer of direct communications and include your contact details, PDF it and send it to your solicitor to pass on. Because this is officially a letter to the seller both solicitors will I think be obliged to pass it on.</p>
Here's a template note to a buyer</a> that worked for me in a previous sale.</p>
Sample letter to buyers</h4>

FAO: [Buyer Name Here]

C/O: [Buyer’s Solicitors]

Ref: [Property Address]

Thu 4 Mar 21</p>
Dear [Name],</p>
This is just a personal note in addition to the formalities around the sale to let you know that we are keen for the process to go smoothly for both sides and that if there’s anything you need from us you are welcome to make direct contact as needed.</p>
All contractual and legal issues will continue to be dealt with through our solicitors and the estate agent, however we have found it useful in the past to be able to co-ordinate directly to ensure a smooth route to a completed sale.</p>
There is no pressure to accept, but should you wish to do so please find my contact details below.</p>
Confirmation of your receipt of this letter would be greatly appreciated.</p>
Yours</p>
[Your Name]</p>
[your.email@example.org]

[01184960000]</p>
</blockquote>
I ain't no lawyer</h2>
Do this at your own risk! Just as this could help, you could also use this communication channel to sink a deal that would have worked out, or even get yourself in legal trouble.</p>


Podcasting hardware setup
2021-01-26T00:00:00+00:00
Current recording/playback setup</h2>

Samson Q2U cardiod XLR/USB Microphone</a> ~ £90

USB means no need for a mixer or XLR adapter thing.</li>
Seems to be the best price/quality balance for me.</li>
The cardiod effect is pretty good, greatly reducing input of noises from elsewhere, especially if you talk really close to it (which I don't).</li>
</ul>
</li>
Proel RSM180 Microphone stand with boom</a> ~ £34

Because standing desk, plus the Samson mic doesn't isolate vibrations from the desk if it's on a desk mount.</li>
</ul>
</li>
AKORD Microphone Swivel Pop Filter</a> ~ £5

Because all the cool kids have one, and it was cheap. No idea if I really need it. Edit: yes I do, breathing on the mics and hard "P"s really do create a bass boom without it as the rush of air hits the mic.</li>
</ul>
</li>
Some old philips bluetooth headphones connected with a real headphone cable to the line out on the back of the Samson mic. I only use this for recording to avoid feedback from speakers. It has a feedback of your own voice which is disconcerting and nice in equal measure. For conference calls zoom and teams seem to do a decent job of cutting out the feedback, and I use mute a lot anyway so I don't bother with the headphones.</li>
The tiny and highly capable Nobsound G3 2 Channel Bluetooth 5.0 Power Amplifier 100W</a> hifi amp, driving a pair of bookshelf Kefs for editing, music and conference calls.</li>
Wharfedale Diamond SW150 Subwoofer</a> hiding at the back, because frequencies below 100Hz</a> are important too.</li>
</ul>
Video</h2>
Camera</h3>

Logitech C920 HD Pro Webcam</a> ~ £60</li>
</ul>
Lighting</h3>

WBTY Gooseneck Mount Stand</a></li>
Neewer USB lights x2</a> ~ £51 (didn't come with usb power units, bring your own, requires high-current capability)</li>
</ul>
Software</h2>

OBS (Open Broadcaster Software)</a>for live streaming + recording video+audio</li>
restream.io</a> for streaming to multiple platforms</li>
Audacity for editing (I streamed a record+edit session</a> if you're interested)</li>
</ul>
Everything else</h2>

Dell XPS15 (sometimes an XPS13)</li>
Linux mint cinnamon</a></li>
Standing desk</a> (no chair these days, previously kneely chairs and exercise balls).</li>
Nextstand k2 Folding laptop stand</a> (amazing bit of kit, very portable too)</li>
Microsoft sidewinder X4 keyboard</a></li>
Logitech M570 trackball</a> - I find the continuous thumb+finger grip needed for a mouse uncomfortable.

Recently upgraded to Logitech MX Ergo</a></li>
</ul>
</li>
HP M477fdw laserjet + sheet feed scanner</a> because life's too short for manually scanning one page at a time. Configured to automatically send scans to server over smb, then sync'd to all devices with syncthing. Win.</li>
</ul>
Pics</h2>
</p>
Also-ran hardware</h2>
I started with a Jabra Evolve 65 wireless headset</a> (~£130 because covid prices) but that heavily processes voice so while it's great for conference calls it's not good for anything beyond your third episode as people will unsubscribe just for the sound quality. Still use them for on the go conference calls and phone calls, and occasionally music (though my phone's bluetooth is currently borked). They eventually also snapped after being thrown in a bag too often, so now I have a...</p>
Jabra Evolve2 65</a> headset, which I use for conference calls (and it's great for that), especially out and about, but which is also too heavily processed to be suitable for recording high quality podcast audio.</p>
I tried a Blue Microphones Snowball USB Microphone</a> (~£75) next, but that picks up everything</em> even in it's alleged cardioid mode. It did make a good recording of some thunder until the kids talked over it in excitement! You could use this if you have a very</em> quiet environment.</p>
Out of my price range</h2>
Some of my friends who take it more seriously splashed out on a Shure SM7B XLR Mic</a> at a mere £389, and you have to by an XLR mixer to run it. I have to say they sound lurvely, and when they knock it you hardly hear it so it's got excellent vibration isolation built in.</p>
You could also look at the Shure MV7 USB Microphone</a> which has USB as well as XLR so you wouldn't need a mixer. That's £250 currently. Presumably it's also got Shure's excellent sound and vibration isolation, though I haven't got one, so let me know if you get one!</p>
See also</h2>

Scott Hansleman's AV kit guide - "good, better, best"</a></li>
</ul>


A book list for my children
2021-01-25T00:00:00+00:00
This is for my children when they're old enough to read them and benefit from them, and when they decide to take an interest in the advice of their boring old man, however I'd recommend this list to anyone no matter their stage in life.</p>
What book should you read next? Well, what problem are you trying to solve right now? That said, there are some eternal problems in life such as interpersonal relationships and the search for financial stability and fulfilment which these books help navigate.</p>
I've added a note to each explaining why you should read it, and when you might want to make it a priority. I've also put them in an order that I think would be sensible; with the most foundational first (personal growth) followed by more wealth and lifestyle focussed volumes.</p>
Being human</h2>

The 7 habits of highly effective people; Stephen Covey</a>

Why? It's important to know you can change who you are and be a better person, and know how to do so. You can be better with relationships and those around you. This book is an inspiration in personal growth.</li>
When to read: as a young adult, and again when you're older and wiser. Or right now if you're already both of those.</li>
</ul>
</li>
Daring Greatly by Brené Brown</a>

Why? Our society teaches all the wrong lessons around guilt, shame and vulnerability. Our default responses to that are self-destructive. Brené shows us how to be brave enough to accept the challenges of life without hurting others to protect ourselves. The difference between guilt ("I did something that was wrong and regret it") and shame ("I'm a bad person") is often misunderstood or ignored entirely. When you think clearly about this it becomes clear that guilt is good and shame is bad.</li>
When to read: As early as you can, and maybe again if you find yourself falling into the traps of self-shaming, shaming others or not having the courage to do hard things for fear of failure.</li>
</ul>
</li>
Nonviolent Communication; Marshall Rosenberg</a>

Why? Somewhat surprisingly perhaps the calm and empathic approach to communication is superior; but that's easier said than done with our brains evolved for times gone by.</li>
When to read: As soon as you argue with someone and wonder what it achieved or it cost you dearly. Ideally before that happens.</li>
</ul>
</li>
Bonds That Make Us Free; C. Terry Warner</a>

Why? Yes, it's another book on dealing with people. The truth is it's not them it's you; but the good news is that you have the power to change what's wrong. It turns out that dealing with people is important, and is foundational in all walks of life.</li>
When to read: As soon as you can, and then probably every decade after that.</li>
</ul>
</li>
So good they can't ignore you; Cal Newport</a>

Why? Don't believe the hype at work, people hire for your real skills and what you can do for them. More details and foundational career advice for any path you choose lie within.</li>
When to read: Probably before your first or second job, but better late than never. If you've retired rich before you read this then maybe don't bother!</li>
</ul>
</li>
The Life-Changing Magic of Tidying; Marie Kondo</a>

Why? Because too much stuff ironically makes you less happy. Go figure. A practical guide to a calming and tidy house/room/boat/mansion/office from someone a little obsessed with tidy.</li>
When to read: When you're fed up hearing me complain your room's a mess. Or later on when you realise why I complained about it but don't really know how to fix it.</li>
</ul>
</li>
48 Laws of Power; Robert Greene</a>

Why? Sadly the world is not the cuddly, friendly, fair and equitable world some people seem to think they can wish into existence by will power and singing alone. There are plenty of good things, but the hard truths about raw underlying human nature are best understood so you can deal with them if you need to. I trust you are a good human being who will only use some of the darker knowledge in here for good and personal survival.</li>
When to read: I hesitate to say early because used without experience this knowledge will get you into more trouble than it gets you out of. That said, earlier the better but don't rush into using it, just observe the people around quietly with this new knowledge for the next few years at least.</li>
</ul>
</li>
Why Has Nobody Told Me This Before?; Julie Smith</a>

Why? We all think we are perfect and we know how to use our brains. Turns out that's hubris, and it's worth having an expert on hand to help you learn to use your own brain properly and communicate in non-destructive ways with those around us. Expert therapy and counselling are great and should be taken advantage of, but it's useful to have a reference and guide to have and to study alone as well, regardless of whether you get expert help.</li>
When to read: as soon as you can, and probably every 5-10 years after that given we tend to forget the lessons and go back to default human patterns.</li>
</ul>
</li>
</ol>
Money and work</h2>

Rich dad, poor dad; Robert T. Kiyosaki</a>

Why? Money isn't everything, but not having it will make you miserable. And the thing that will make you poor beyond anything else is your mindset and financial education. Read this to learn what the people who never seem to work do and why they never seem to run out of money.</li>
When to read: As soon as your bank account hits zero for the first time, and any time you are annoyed that you still have that sucky job you hate but can't escape from (happens to us all I think).</li>
</ul>
</li>
Think and Grow Rich; Napoleon Hill</a>

Why? More money will let you solve more problems, have more freedom, help more people, and live your best life. But it's not that easy to know where to start without a role model and a plan. Good practical guidance in here. Time will tell if I make good use of it myself.</li>
When to read: After the first few years of plying a trade perhaps, but don't leave it too long.</li>
</ul>
</li>
The Millionaire Fast Lane; M.J. DeMarco</a>

Why? Having read a lot of books about business and money, none of them have the completeness of this one. It's the best guide to the whole picture I've come across.</li>
When to read: As soon as you start trying to earn your own money, or as soon after that as you can.</li>
</ul>
</li>
The 4-Hour Work Week; Tim Ferris</a>

Why? Working for someone else 9-5 isn't the only way to make a living. This book might age when it comes to tactics, but the inspirational vision of another way of life is timeless.</li>
When to read: Whenever you've had enough of a job. Or maybe if you just want to dream, or travel.</li>
</ul>
</li>
The Road Less Stupid; Keith J. Cunningham</a>

Why? Whatever money and success you achieve, it's very easy to throw it all away again by doing dumb things that seemed like a good idea. This is a straight-talking guide to not doing that. Contains important lesson about protecting against downside risk (i.e. a small chance you lose everything) when chasing upside risk (i.e. a chance to be much richer etc). Unmitigated downside risk has the power to destroy everything you've worked for.</li>
When to read: Sooner rather than later, especially if you're feeling a bit ambitious.</li>
</ul>
</li>
Working Identity; Herminia Ibarra</a>

Why? The days of job-for-life are long gone, even the days of one-career lives are fading. You'll almost certainly need to evolve your work life, if not blow it up completely and start over. This book is a wise guide to the opportunities and pitfalls of seeing if the grass really is greener.</li>
When to read: when you are sick of your job/career and want to do something different.</li>
</ul>
</li>
</ol>
The meaning of life, evolution and religion</h2>

The Blind Watchmaker; Richard Dawkins</a>

Why? Evolution if often poorly understood. As the foundation of our existence and behaviours this to me is a must read to truly understand life. For me it answers the question of "what is the point in life"; ironically there is no point really, but that is more freeing than some deity with odd morals. Just know that you can know the raw mechanics and still live a full, happy and meaningful life; helping all those around you.</li>
When to read: Whenever you have some time. Kinda background knowledge for life.</li>
</ul>
</li>
The God Delusion; Richard Dawkins</a>

Why? God is all around us in the many people's of the world. His name(s) are often used to manipulate and exploit, and play power games. You and I can probably do nothing about this (and it's dangerous territory to tread in, I'm surprised Dawkins is still alive), however I think the knowledge is an important piece of the complex puzzle that is humanity. Once you've read this, don't judge those who believe for whatever reason, you will not change them and you will only cause hurt.</li>
When to read: Whenever you have time, any time you are considering the meaning and truth of religion, and any time you want to be able to discuss the topic with more useful information.</li>
</ul>
</li>
</ol>
Children of your own</h2>

Pregnancy For Men</a>

Why? Nothing can prepare you for this experience, and men are an anxious side-show. This book helped me cope, and even occasionally be useful.</li>
When to read: As soon as you think you'll have a kid on the way, don't leave it till the due date or you'll miss the pre-natal stuff.</li>
</ul>
</li>
How to talk so Kids Will listen</a>

Why? Nothing about communicating well is natural or obvious. It takes work and understanding, and undoing cultural failings. This series is one of the best, kindest works out there, and helped me immensely.</li>
When to read: When your kids are starting to talk.</li>
</ul>
</li>
Hold on to Your Kids: Why Parents Need to Matter More Than Peers</a>

Why? Culture is teaching us to ignore our parents with devastating consequences for all involved. It turns out "attachment" is the key, and when it goes wrong nothing you do will work.</li>
When to read: When your kids are 4+ years old.</li>
</ul>
</li>
</ol>

Personally I like the audiobook format, and at time of writing a yearly subscription to Audible was good value for getting easy access to the content, and a narration from the author or a good speaker makes for a more memorable experience.</p>
Beyond this list, check the best-seller lists for whatever subject you wish to excel at. There's no point learning the hard way when you can learn from someone who has. Even in this digital age books still pack a tremendous punch; the result of someone pouring 20+ years of their experience into something you can read in a week is like liquid gold. I also have recorded a lot of my personal reading list on GoodReads</a>.</p>
To my little people who will soon be bigger than me in every way, love you 💕️👊️, Diddy x</p>
P.S. If you're reading this and you're not my kid, well I love you too for being the lovely human I know you are.</p>


Remote code interview pairing tools
2020-12-04T00:00:00+00:00
Things that you could use to do simple coding exercises in the new world of remote everything.</p>

CodeInterview.io</a> - $5/interview or $55/month</li>
CodeSandbox.io</a></li>
Cyber-dojo</a> - free for non-commercial, set your own price for commercial!</li>
VSCode live share</a> works well, it can do voice as well.</li>
Zoom/hangouts/meet/skype etc. built-in screen sharing and whatever coding setup they already use.</li>
</ul>
Thanks to my friends at eSynergySolutions for the suggestions, recorded here because the internet has a better memory than me.</p>


Database migration options for dotnet core
2020-11-26T00:00:00+00:00
Options</h2>

Built-in EF Core migrations</a></li>
FluentMigrator</a></li>
DbUp</a></li>
RedGate Deploy</a></li>
RedGate SQL Change Automation</a></li>
Flyway by RedGate</a></li>
SqlHawk</a> (I wrote this years ago, forked from the excellent SchemaSpy</a>)</li>
? hit me up on twitter</a> if you know of any more options</li>
</ul>
Note that the excellent ReadyRoll got bought by
RedGate</a>
and renamed to something I can never remember.</p>
See also</h2>

https://stackoverflow.com/questions/46649238/alternative-to-redgate-readyroll-yet</a></li>
</ul>


Running the CosmosDB emulator on Linux
2020-09-17T00:00:00+00:00
Update: microsoft have released a linux compatible docker image it seems, let me know if it actually works and I'll update this post. https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator</a></p>

Unfortunately even though "Microsoft 🖤 Linux" the CosmosDB team seem less keen. This is not fun, pretty or stable, but at least it works which is something.</p>
There is a docker image but it only runs on docker-for-windows so it's no help here.</p>
Overview</h2>

Grab a Windows VM, install the emulator.</li>
Copy the emulator SSL cert to the host.</li>
Map the ports.</li>
Redirect the VM IP back to localhost with iptables.</li>
</ol>
Steps</h2>
Get a Windows VirtualBox VM Running</h3>
Download the VirtualBox image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/</a> (2 month free trial, no key needed).</p>
Install VirtualBox.</p>
Fix the VM image configuration (if you are using the slightly older VirtualBox from ubuntu's apt repositories) by removing controller="VBoxSVGA"</code> from the .ovf</code> file in the .ova</code> file (which is a tar.gz archive). See https://schnouki.net/post/2020/how-to-run-a-win10-dev-vm-on-virtualbox-5/</a> for details.</p>
Import the VM into VirtualBox</p>
Run the VM</p>
Install the guest additions (Devices > Insert guest additions CD)</p>
Set the timezone (VM defaults to US).</p>
Turn on seamless copy-paste & drag-and-drop (so that we can move certificate and data files to/from the host easily).</p>
Set up the CosmosDB emulator</h3>
In the Windows VM download and install the CosmosDB emulator in the VM from https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator?tabs=ssl-netstd21#installation</a></p>
If the emulator is already running, shut it down:</p>
cd </span>"</span>C:\Program Files\Azure Cosmos DB Emulator</span>"</span></span>
.\</span>Microsoft.Azure.Cosmos.Emulator.exe</span> /</span>shutdown</span></span></code></pre>
Start the emulator with network access enabled.</p>
cd </span>"</span>C:\Program Files\Azure Cosmos DB Emulator</span>"</span></span>
.\</span>Microsoft.Azure.Cosmos.Emulator.exe</span> /</span>AllowNetworkAccess </span>/</span>DisableRateLimiting </span>/</span>Key</span>=</span>'</span>C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==</span>'</span></span></code></pre>
(This key is the hard-coded well-known key that the emulator uses by default, which makes it easy to connect the storage explorer).</p>
Check it is listening on 0.0.0.0:8081</code> (as opposed to 127.0.0.1:8081</code>):</p>
netstat -a |findstr 8081</span></span></code></pre>Map the CosmosDB port to the host</h3>
In the VirtualBox network settings for the VM, click advanced and add a port mapping for 8081 on the VM to 8081 on the host.</p>
See https://www.howtogeek.com/122641/how-to-forward-ports-to-a-virtual-machine-and-use-it-as-a-server/</a></p>
This makes the emulator appear as if it was running on the Linux host.</p>
(Note there are replica ports that need mapping too, see the dotnet-core section below).</p>
Connect with Storage Explorer</h3>
Install the Azure Storage Explorer on your Linux host, either with the .tar.gz or from the snap store</p>
https://azure.microsoft.com/en-us/features/storage-explorer/</a></p>
Run the storage explorer.</p>
Connect to the emulator on localhost.</p>
You should now be able to create collections and add documents, and then view them.</p>
Connecting from dotnet-core</h3>
Problems connecting from core that I didn't encounter with storage explorer:</p>

The SSL certificate is rejected.</li>
CosmosDB sets the ReadEndpoint</code></a> to be the VM's IP address, which isn't accessible from the host because we are using NAT and port mapping. (oddly this wasn't consistently the interal IP address.</li>
The replicas listen on additional ports which also need mapping.</li>
</ol>
Accepting the emulator SSL certificate</h4>
This is the error you get trying to connect to the CosmosDB emulator from dotnet core on Linux without adding the certificate to the Linux host's certificate authority (CA) list:</p>
System.Net.Http.HttpRequestException:</span></span>
The SSL connection could not be established,</span></span>
see inner exception. ---> System.Security.Authentication.AuthenticationException:</span></span>
The remote certificate is invalid according to the validation procedure.</span></span>
  at at System.Net.Security.SslStream.StartSendAuthResetSignal(</span></span>
	ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)</span></span></code></pre>
Export the emulator's public SSL certificate "DocumentDbEmulatorCertificate" as base64 X509.</p>
Reference: https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator-export-ssl-certificates#how-to-export-the-azure-cosmos-db-tlsssl-certificate</a></p>
Copy the exported file to your Linux host.</p>
Add the certificate to the trusted certificates on your machine (these instructions are for Ubuntu/Mint):</p>
sudo cp --verbose cosmos-emulator.cer /usr/local/share/ca-certificates/cosmos-emulator.crt</span></span>
sudo update-ca-certificates --verbose</span></span></code></pre>
Reference: https://stackoverflow.com/questions/44159793/trusted-root-certificates-in-dotnet-core-on-linux-rhel-7-1/44160125#44160125</a></p>
Note the change in file extension from .cer</code> to .crt</code> between export and import (I'm not 100% sure if this matters).</p>
Note that the certificate is regenerated on emulator startup</em> so you'll either have to do this every time or pass the emulator a predefined certificate to use.</p>
If you want to validate the certificate has been installed, see https://unix.stackexchange.com/questions/97244/list-all-available-ssl-ca-certificates</a></p>
Redirecting ReadEndpoint traffic</h4>
In the VM in a powershell or command prompt run ipconfig</code> to get the machine's IP address.</p>
Tell iptables</code> to redirect all requests for the VM's IP address back to localhost</code>:</p>
sudo</span> iptables</span> -</span>t</span> nat</span> -</span>I</span> OUTPUT</span> -</span>-dst</span> 10.0.2.XXX</span> -</span>p</span> tcp</span> -</span>-dport</span> 8081</span> -</span>j</span> REDIRECT</span> -</span>-to-ports</span> 8081</span></span></code></pre>
Where XXX</code> is the IP of your VM.</p>
As per: https://unix.stackexchange.com/questions/441182/how-to-map-an-ip-address-to-localhost</a></p>
Mapping replica port(s)</h4>
On the VM run netstat -a |findstr LISTENING</code></p>
You'll see a bunch of ports in the 10,000-20,000 range. These are the replicas</p>
I found out which ports to map by running the client code and looking at the exception when it failed to connect. It was only port 10253</code> for me so far. Here's the error for a failure:</p>
Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.</span></span>
ActivityId: 2dfaf5dc-0d5a-4ed7-828d-c16cdcd117a3, </span></span>
RequestStartTime: 2020-09-17T20:00:16.8868999Z, RequestEndTime: 2020-09-17T20:00:46.7777188Z, Number of regions attempted: 1</span></span>
ResponseTime: 2020-09-17T20:00:16.9400284Z, StoreResult: StorePhysicalAddress: rntbd://10.0.2.15:10253/apps/DocDbApp/services/DocDbServer15/partitions/a4cb495b-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: True, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-09-17T20:00:16.9221735Z, activity ID: 2dfaf5dc-0d5a-4ed7-828d-c16cdcd117a3, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D], URI: rntbd://10.0.2.15:10253/, connection: <not connected> -> rntbd://10.0.2.15:10253/, payload sent: False, CPU history: not available, CPU count: 8), ResourceType: Document, OperationType: Query</span></span></code></pre>
Add these ports to the VirtualBox port mapping configuration for your VM.</p>
In the VM in a powershell or command prompt run ipconfig</code> to get the machine's IP address.</p>
Tell iptables</code> to redirect all requests for the VM's IP address back to localhost</code>:</p>
sudo</span> iptables</span> -</span>t</span> nat</span> -</span>I</span> OUTPUT</span> -</span>-dst</span> 10.0.2.XXX</span> -</span>p</span> tcp</span> -</span>-dport</span> 10253</span> -</span>j</span> REDIRECT</span> -</span>-to-ports</span> 10253</span></span></code></pre>
Where XXX</code> is the IP of your VM and 10253</code> is the replica port.</p>
As per: https://unix.stackexchange.com/questions/441182/how-to-map-an-ip-address-to-localhost</a></p>
Importing data</h2>
If you need to import data then use data migration tool from https://docs.microsoft.com/en-us/azure/cosmos-db/import-data#Install</a></p>
Success</h2>
With all of the above I was then able to connect a dotnet-core application on Linux to a CosmosDB emulator.</p>
Emulator failures</h2>
After a while and some unrecorded fiddling, the emulator then gave up the ghost, resulting in the Azure Storage Explorer reporting the apparently meaningless error message "Unable to retrieve child resources."</p>
According to the internet this error could mean anything. https://social.technet.microsoft.com/wiki/contents/articles/53393.azure-storage-explorer-troubleshooting-unable-to-retrieve-child-resources-or-the-request-action-could-not-be-completed.aspx</a></p>
I might just blow the whole VM away and start again...</p>
Should you use CosmosDB?</h2>
CosmosDB comes with significantly increased costs:</p>

I gather it's very expensive to run in production</li>
Development is painful</li>
</ul>
As a result you should only choose CosmosDB for your solution if you really</em> need something it offers that you can't get from a more straight-forward SQL or DocumentDB solution and your domain is a good match for the document-database (aka NoSQL) model.</p>
Further reading</h2>

Azure Development using CosmosDB offline Emulator by Matt Thankachan</a></li>
</ul>


Effective GTD with Trello
2020-09-15T00:00:00+00:00

Learn how I've refined the GTD process with Trello for modern living.</li>
Get a template GTD board to kick-start your own.</li>
</ul>
GTD+Trello has really helped sooth my neurotic tendencies around todo lists. I'm often startled by starting off thinking I have more than I could ever achieve and then getting all my top priorities done in one sitting and being able to truly relax afterwards. No more nagging feeling there's something urgent at the bottom of the pile of papers somewhere.</p>
What is GTD?</h2>
Getting Things Done https://en.wikipedia.org/wiki/Getting_Things_Done</a> is an old but effective method for organising the flood of todos that enter your life in many forms. We need it now more than ever in our infinitely connected world. And as a middle-aged working parent I need it now more than I ever did.</p>
The point of GTD is to eliminate all the other places that you might have hidden todo lists such as</p>

unread emails</li>
your memory</li>
phone notes</li>
bits of paper</li>
phone screenshots</li>
other unstructured todo lists (Trello, Evernote etc. let me know what you use now that isn't working for you and I'll add it to the list)</li>
</ul>
And then to make your todo list something that you can almost mindlessly churn through, eliminating huge task switching costs caused by badly defined and variably sized tasks.</p>
Trello + GTD</h2>
The original GTD method involved bits of paper. But now we have Trello, which is an excellent fit for GTD with its cards, lists and many integrations. There's a great write-up of using Trello for GTD here: https://blog.trello.com/gtd-getting-things-done-maximizing-productivity-trello</a> which I suggest you go and read first. It's how I got started, however I found I was still left building my own process and set of lists from there before it really worked for me.</p>
Trello is great for GTD because:</p>

You can set up lists how you like</li>
There are apps so you have it everywhere</li>
You can forward emails to it (yay, inbox zero at last)</li>
It supports attaching pictures, PDFs etc.</li>
You can add checklists, due dates, date-stamped comments etc.</li>
Drag-n-drop sorting and moving of cards</li>
Buttons for archive / move on the cards</li>
A lovely cross-platform interface</li>
It's free!</li>
</ul>
Disclaimer</h2>
Neither GTD or Trello can give you more hours in the day, help you focus, get you to do things you really don't want to do, or make your tax return go away. This still requires putting the work in. Having said that, at least you will know you haven't just forgotten anything really important, it's just a bit stuck on your list. Personally with life as it is for me now I never get close to the weekly/daily GTD schedule laid out below, but that doesn't make it less valuable.</p>
If you are looking at GTD for the same reasons as I did then beware "the efficiency trap"</a></p>
My GTD Method</h2>
Here's my actual GTD board (minus details) to give you an idea of what it looks like. You might be able to see I have far too much in Inbox and Action items as is often the case, but it doesn't take too long to clear down when I actually focus on it.</p>
</p>
First you need to get everything you might have to do into your GTD board. Don't think too hard at this point, just ping everything to Trello and deal with it efficiently at your next GTD moment:</p>
Getting Everything to the Board</h3>
Emails</h4>
Forward emails to your GTD board "send-to-board" email address to get to inbox zero quickly. Make sure your GTD board email address is in your address book so you can just type "gtd" and hit send.</p>
</p>
Using GTD instead of your unread email as a todo list makes inbox-zero possible again.</p>
The subject line becomes the card title so edit that if you want. The body of the email becomes the card description so you can add to that too.</p>
Top tip: When sending an email that you'll need to follow up on, put the GTD email address in the Blind-Carbon-Copy (BCC) field so that you instantly get a copy in your GTD board with no extra faff. (Thanks to this podcast episode for that idea: "Organizing the LEGOs In Our Minds to Build the Ultimate Version of Ourselves with Tiago Forte - Flavor of Leadership"</a>).</p>
Web based tasks</h4>
Install AddToAny</a> to send any webpage from Firefox to your GTD inbox.</p>
</p>
Documents, physical objects, pictures, screenshots</h4>
Install the Trello app and use the native share buttons to share pictures, web pages etc to your GTD inbox. Use this to take photos of paper documents and physical things you need to deal with and send the picture straight to your GTD inbox on Trello.</p>
</p>
Verbally Received Tasks</h4>
For when someone just says to you "oh, can you do this for me?" and you don't want to be that person that lets them down by forgetting.</p>
If you think you are "the forgetful type" (you aren't, that's just a negative self-label compounded by habits), then you can fix it forever by just being more organised, no-one will ever know it was GTD, they just think you remembered for them.</p>
Open up Trello on your phone or laptop and add a card. Or if it's easier write an email to your GTD board!</p>
Using Alexa/Siri etc</h4>
For advanced magic teach your Siri/Alexa/Google to email to GTD by mere voice command</p>

Hey google, email gtd ...</p>
</blockquote>
Or maybe set up a custom action. (If anyone's done this send me some details and I'll expand out this section. PRs welcome!)</p>
GTDing</h3>
Inbox Processing (Daily)</h4>
Go through inbox cards and triage them as follows:</p>

Unactionable or not important?

Archive the card</li>
or move to reference</li>
or move to someday.</li>
</ul>
</li>
Less than 2 mins?

Do now.</li>
</ul>
</li>
For someone else?

Move to "waiting"</li>
</ul>
</li>
Need to just mull it over a bit?

Move to "incubate"</li>
</ul>
</li>
For future time/date?

Add to google calendar</a> as event or timed reminder and move to "scheduled".</li>
</ul>
</li>
More than single step to complete? (Even two really small steps counts)

Move to "projects"

and add a checklist.</li>
For some multi-step things I keep it as a single card, put it in actions but put the next step in the title (e.g. "something to do - call bob"), and when I've done that step I update the title to the next step. It's a bit less overhead than juggling multiple cards, but it's a judgement call whether it's worth a project card. I like project cards for things that are going to drag on over months.</li>
</ul>
</li>
</ul>
</li>
Finally if none of the above then:

move to "action items"

and edit the card to make sure it's doable in one go (add links, forms, phone numbers etc).</li>
</ul>
</li>
</ul>
</li>
</ul>
I've learned to do more of this in the Trello app when I have five idle minutes during the day so that when I sit down at the laptop I don't use all the time just getting GTD straight.</p>
I add some Trello labels to the cards so that I can tell whether a card is something I can pick up in any particular situation (do I need the laptop? good phone signal? good internet? do I need to be somewhere in particular?) The situational ones are all black with an @</code> prefix, e.g. @home</code> or @good-internet</code>.</p>
Projects (Weekly)</h4>
Go through each project</p>

Do I still need this project (move to someday, or just archive the card if not).</li>
Create next action card - use the "convert checklist item to card" feature of Trello.</li>
</ul>
Waiting (Weekly)</h4>

Set due dates on the cards so that they'll get highlighted if they are due chasing next time you  review.</li>
Chase up if no action from others and past the due date (maybe fire off an email or WhatsApp).</li>
Archive if dealt with, or move back to inbox if it's thrown up other issues.</li>
</ul>
Rules - Thou Shall Not Scroll</h4>
I.e. things that I've learned knock me off track when trying to do a "GTD" session.</p>

DO NOT READ the articles - send them to Pocket</a>.</li>
DO NOT READ the newsletters - leave them as unread in mail inbox.</li>
NO twitter/facebook/linkedin FEED reading (posting is okay).</li>
</ul>
Dealing With Action Items</h3>
Once your GTD board is triaged your inbox should be empty and your action items should be prioritized and full of things that you can "just do" without getting blocked. Start at the top and work down as if your boss told you to get on with it.</p>
This is where the magic happens!</p>
What to do when "Action Items" is too long or not getting done</h3>
I found that after a while Action items gets overwhelming. Cards end up stuck in there forever, never quite getting to done.</p>
To deal with this, use the "move all cards in list feature of Trello".</p>
You have two choices here depending on the state of your list and how important thing things in it are:</p>

Move all cards back to Inbox and re-triage them.</li>
Move all the cards to "Someday" because they actually weren't that important.</li>
</ul>
Or you might pick a few to bump back to inbox then move everything else.</p>
This gives you a complete reset, and eliminates the mental overwhelm that results from an unrealistic todo list.</p>
You might initially think this is a failure of the GTD method, but actually without GTD you are just forgetting things that have been on your brain's todo list, and maybe some of them were actually important. With this approach you have the opportunity to decide what you want to punt to the eternal pot of "someday" with no angst that you are dropping important tasks.</p>
If you have tasks that really are important and they are never getting done, then</p>

you're avoiding it and you need to move them to the top and just do them, a well formed action card is often not as bad when you actually tackle it,</li>
or you don't allocate sufficient time in your life for these tasks, and it's up to you if you want to adjust your schedule to get more done.</li>
</ul>
Watch Out For</h3>
Spending all your time just arranging your GTD board and never doing any of the action items. Are you avoiding an unpleasant task? Are your tasks too big and need breaking down?</p>
When the lists get long</h3>
You have to make sure you're investing enough time on average for the piles not to build up and up.</p>
When my list got to the several hundred range I needed to know for my own sanity whether I was winning or losing week by week, it's hard to remember whether the inbox was on 197 or 185 the week before. Being a bit of a systems person not afraid of a spreadsheet, I enabled the firefox trello card counter addon</a>, manually copied them into a spreadsheet, added a graph, and now I can see if I need to spend more or less time getting on top of it.</p>
</p>
</p>
I've shared a google sheets copy of the card count graph spreadsheet</a> which you can use yourself.</p>
Learn more</h2>
Ready for Anything Book</h3>
David Allen wrote a follow-up book with a huge amount of wisdom and experience embedded in it. This one is like having a management consultant in your ear guiding you to be the CEO of your life and work. Amazing. Not to be missed once you have the basics down.</p>
"Ready for Anything: 52 Productivity Principles for Work and Life" by David Allen</a></p>
Podcast</h3>
Having run this system for a while, it grew to a daunting few-hundred cards in inbox/actions. To figure out what I'm doing wrong I did some more research.</p>
The official GTD podcast</a> is gold, hearing from the creator and his consultants on how they really use it and help clients get unstuck was really helpful.</p>
Key points I learned:</p>

Projects are not what you'd normally think of, they're just anything bigger than one action. Anything</em>. (You can add another layer for big projects if that's your world).</li>
The weekly review isn't just shuffling your todo pile. It's stepping back and considering the big picture, setting strategic direction, and just having time to reflect.</li>
If tasks are getting stuck, or you are avoiding them it's probably because you haven't taken the time to step back and think clearly about whether it's something you actually want to do (someday/incubate/delete), and if you do then thinking about what the one next step might be (e.g. "google xx to learn more" about it).</li>
</ol>
Articles</h3>
In no particular order, and of varying usefulness:</p>

https://facilethings.com/blog/en/starting-gtd-when-you-are-busy</a></li>
https://lifehacker.com/the-weekly-review-how-one-hour-can-save-you-a-week-s-w-5908816</a></li>
https://todoist.com/productivity-methods/getting-things-done</a></li>
https://blog.doist.com/gtd-tips/</a></li>
https://www.reddit.com/r/gtd/comments/8bv3as/how_do_i_keep_my_gtd_inbox_from_overflowing_to/</a></li>
https://plan.io/blog/getting-things-done/</a></li>
https://www.taskade.com/blog/outliner-note-taking/</a></li>
https://www.taskade.com/blog/getting-things-done-gtd/</a></li>
https://www.taskade.com/blog/hierarchical-note-taking/</a></li>
https://zapier.com/blog/gtd-getting-things-done/</a></li>
</ul>
Get started with a template GTD+Trello board</h2>
You can easily create your own GTD Trello board from this template</a> that I've created for you.</p>
The template contains:</p>

Preconfigured columns,</li>
Labels,</li>
Some example cards,</li>
A full instruction list in the inbox for daily processing,</li>
A butler action button for moving to "someday".</li>
</ul>
I reckon using this ready-made template will save you at least an half-an-hour of your time just fiddling with Trello to get going. It'd also be a nice way of saying thanks if you found this article useful.</p>
</p>
Beyond GTD</h2>
I still struggle with what to do today</em> even with GTD in place. I've found the (paid) Sunsama</a> app to be a game changer, especially as I transition to full business owner from sole contractor. You could do it without, but having an app that continuously reminds you what you planned to do today & this week makes sticking to priorities so much easier.</p>
</p>


Approaches to refactoring, technical debt and legacy code
2020-07-09T00:00:00+00:00
Sometimes a codebase has an overwhelming amount of "terrible" stuff that as a developer you almost can't help but just diving in and fixing it. Doing this without thinking too hard can result in many variations of failure, such as:</p>

Upsetting the people paying you because you are doing something they don't consider important.</li>
Never finishing the mammoth rewrite you took on, resulting in a mess of two different styles of code.</li>
Personal burnout.</li>
Fixing things that maybe weren't actually that important (even if they make you cry whenever you have to look at their source code), at the expense of failing to fix more urgent problems or building useful features.</li>
</ul>
This article is a collection of research, links, thoughts and ideas that I've captured here for both of our benefits. It's a bit disjointed to read, but the point was more to capture all the key concepts and resources that are floating around in tech land.</p>

⏩ If you want the executive summary, head over to my Leaders guide to technical debt - aka "why can't we ship anything!?"</a> 🧐. For the technical details read on below.</p>
</p>
Must-read articles</h2>
Here's a couple of must-read articles on the subject:</p>
Refactoring</h3>

"We take the next feature that we are asked to build, and instead of detouring around all the weeds and bushes, we take the time to clear a path through some of them."</p>
~ Ron Jeffries</p>
</blockquote>
In short, don't make debt cleanup its own task, just tackle what gets in your way as you go, and leave the camp tidier than you found it.</p>
I recommend reading the whole article here: Refactoring -- Not on the backlog! by Ron Jeffries</a>, it's not too long.</p>
Technical Debt</h3>

"Tech debt": an overloaded term. There are at least 5 distinct things we mean [when] we say “technical debt”.</p>

Maintenance work</li>
Features of the codebase that resist change</li>
Operability choices that resist change</li>
Code choices that suck the will to live</li>
Dependencies that resist upgrading</li>
</ol>
</blockquote>
Read the full explanation of each type of debt here: Towards an understanding of technical debt by Kellan Elliott-McCrea</a>, a bit longer but an important piece of writing.</p>
Huge piles of debt</h2>
If your project is sooo "bad" that you feel like throwing it out, you might do well to heed this quote:</p>

"There is only one way to eat an elephant: a bite at a time."</p>
~ Desmond Tutu (via Psychology today of all places!)</a></p>
</blockquote>
Which to me means iterate your way to good.</p>
It doesn't mean have no plan, in fact you should know where you are trying to get to and how realistic that is so you don't end up with a rewrite that can never be completed because it was too ambitious.</p>
Even more reading</h2>
Is a mess a debt?</h3>

"A mess is not a technical debt. A mess is just a mess."</p>
~ Uncle Bob</p>
</blockquote>
A Mess is not a Technical Debt by Uncle Bob</a> suggests that the use of the term debt is for a considered short term trade-off just like taking out a loan, but a mess is nothing of the sort as there is no up-side to creating a mess versus having a clean but temporary solution to a problem.</p>
More taxonomy of bad code: the Reckless/Prudent vs Deliberate/Indavertent quadrant</h3>
Martin Fowler shows that you can categorize debt by whether it is reckless or prudent, and separately whether it was deliberately or inadvertently added to the code.</p>

"Technical Debt is a metaphor, so the real question is whether or not the debt metaphor is helpful about thinking about how to deal with design problems, and how to communicate that thinking. A particular benefit of the debt metaphor is that it's very handy for communicating to non-technical people."</p>
"The useful distinction isn't between debt or non-debt, but between prudent and reckless debt ... there's also a difference between deliberate and inadvertent debt."</p>
"The decision of paying the interest versus paying down the principal still applies, so the metaphor is still helpful for this case."</p>
~ The Technical Debt Quadrant by Martin Fowler</a></p>
</blockquote>
These are useful definitions that can help clearly communicate complex issues with the code to people not directly involved in the code.</p>
It's well worth reading much more of Martin Fowler's writing. There's so much to learn from Martin about programming good practice; the articles are all written in a very human and accessible style.</p>
Legacy tests</h3>
Now that TDD is widely adopted we're faced with cleaning up the output of those who cargo-culted</a> test coverage, or had well meaning but ultimately badly formed attempts at adding tests to their code, sometimes with prodigious volumes of test code.</p>
I think "Legacy tests" is a useful term to describe these problematic tests. Nat Pryce outlines some useful hints that you have legacy tests on your hands:</p>

"Symptoms of legacy tests I have encountered include:

...

Tests are named after issue identifiers in the company's issue tracker.

Bonus points if the issue tracker no longer exists."</p>
~ Nat Pryce, Working Effectively with Legacy Tests</a></p>
</blockquote>
Prevention is better than cure</h2>
How do we stop it building up in the first place?</p>
I see several sources of the build up of technical debt:</p>

Lack of knowledge and experiencd of programming patterns and practices to keep debt in check. This can be addressed with courses, learning from peers, attending meetups and conferences, reading books, watching youtube and many other ways of getting input from beyond the current code editor.</li>
Problematic mindsets in programmers - short term thinking, narrow thinking, or blindness to the broader consequences of the choices being made in the moment. This is a more stubborn malaise, and people are not quick to change their identity. Some will respond well to just being asked to consider the second and third order effects of their choices (e.g. "you could give this other team direct access to the sql db instead building an API for them, and yes it's quicker, but then you won't ever again be able to rename a column without consulting with them), but some people will double down in their thinking and are not open to improvement.</li>
Dysfunctional dynamics between engineering and their non-technical peers and leaders. Common issues include:

Deadline-driven-development (especially when unrealistic and vigorously enforced),</li>
Asking non-technical leaders to make engineering decisions.</li>
</ol>
</li>
</ol>

The client / boss / project manager responds [whilst thinking “I have no idea what you are talking about, or why you are asking] “Which is quicker? Do that one.”</p>
Then the engineer gets the hump that they can’t do a good job.</p>
~ from "Why do automated tests matter?" by Tim Abell</a></p>
</blockquote>
The audio version from Codurance</h2>
Codurance</a> hosted an insightful round-table podcast episode with a group of people who are clearly very experienced. You can listen here: https://codurance.com/podcasts/2019-01-21-legacy-code/</a> and will doubtless be inspired by some things in there. The conversation takes a little while to build momentum but it's worth the wait.</p>
Here some highlights of what I learnt from listening to the show:</p>
The Feathers Book</h3>

Working Effectively with Legacy Code by Michael C. Feathers</a> is the</em> book to read on the subject.</li>
</ul>
Feathers defines legacy code as code without tests.</p>

"Preserving behaviour is a large challenge. When we need to make changes and preserve behaviour, it can involve considerable risk."</p>
~ Michael C. Feathers</p>
</blockquote>
This book is heavily geared towards unpicking the difficulties in getting untested code under test in object-oriented static typed languages such as Java, C# and C++, and modifying it safely.</p>
It is a detailed and thorough treatment of all the tactics available to bring untested object-oriented code under test, specifically sprawling and smelly classes and methods. It won't help you with larger scale problems such as poor structuring of microservice architecture, message bus systems etc.</p>
This article summarizing the Feathers' Legacy Code book</a> has some of the tactics detailed and gives a good flavour of the book so you can decide whether to dive deeper.</p>
Named techniques and related libraries</h3>

Characterization testing</a> is the idea of creating tests to probe and demonstrate the existing behaviour of previously untested code.

Related to this is "approval tests" which allow you to easily incorporate snapshots of output (json, xml, logs etc) into your tests in order to capture existing behaviour and be able to spot any variations that pop up during refactoring.</li>
ApprovalTests.net</a> is a dotnet library for implementing approval tests.</li>
</ul>
</li>
Introducing seams</a> into software can be a useful technique for breaking down untestable monoliths into testable chunks on the way to better code.</li>
Mutation testing (wikipedia)</a> (more info on mutation testing at csharp academy</a>) is a useful way of checking how good your test coverage really is. It is the idea of making (almost) random changes to the code under test to see what whether your tests spot the change in behaviour.

For dotnet this can be done with Stryker.net</a></li>
</ul>
</li>
</ul>
Approaches from hard-won experience</h3>

Make as few changes as possible to get untested production code under test. The first cut of tests will likely be fragile.</li>
It's more important that legacy code that is already in production continues to behave as it currently does than that it behaves as originally specified. People and downstream systems may now rely on that "incorrect" behaviour.</li>
Does the organisation (culture, systems, pressures etc.) cause bad code to be created? If you don't fix that then you will always get more "legacy" code.</li>
The importance of competent technical leadership within an organisation for preventing the build up of catastrophic levels of technical debt.</li>
When communicating, quantify the cost of problems with the legacy code. E.g. "you are losing 1 in 5 developer-days to coping with bugs introduced due to the lack of automated regression tests".</li>
Have the hard conversations with the business about the cost of fixing the mess.</li>
Doing a rewrite is (almost) always the wrong answer.</li>
Get small wins, even if you are facing a huge challenge.</li>
</ul>
Maybe modelling is the problem</h2>
Something that looks like bad code could be that way because of a failure to properly model the real world.</p>
Domain Driven Design (DDD) has much to teach on the matter, and this is a great video on the why modelling could be the problem:
Technical debt isn't technical - Einar Høst - DDD Europe 2019 - YouTube</a></p>
Small Batches Automated Testing Podcast Episode</h2>
Podcast episode Software Delivery in Small Batches: Automated Testing with Jason Swett</a> talks about practical approaches to legacy code, including the idea that you actually can't always avoid writing some more untested code because of the cost-benefit trade-off.</p>

"You can’t just slap a test on any change that you make because the underlying functionality might be really complex"</p>
</blockquote>
The episode describes the evolution of our industry in learning the importance of test coverage for avoiding known risks in software development, and declares that you cannot truly call yourself a professional developer if you are not writing tests.</p>

"I always recommend starting with the easiest most trivial stuff, adding some tests in those areas and then working up to the bigger areas"</p>
</blockquote>
Some relevant thoughts from the episode:</p>

Good test coverage is a pre-requisite to any higher-level automation such as continuous-deployment. (1:35)</li>
To add tests to a project that has no coverage you have to retrofit test tooling and infrastructure (CI, data management etc). "That can be very non-trivial, even for someone very experienced, ... it's going to be a huge challenge to add that stuff retroactively." (3:10)</li>
With regards to the idea of "Declaring that we are going to write tests for every PR from now on." (7:10)

"You can't just slap a test on any change that you make because the underlying functionality might be really complex. In order to write a test you might have to have certain setup data to create the state required for that test might be extremely complicated. To expect somebody to include a test with every PR is not realistic. For some features your change might be 10 minutes of work but to get the test setup, that might be weeks of work in order to take the 20-minutes to write the test."</li>
"I always recommend starting with the easiest most trivial stuff, adding some tests in those areas and then working up to the bigger areas. It seems that you should start with the most important stuff but that's not realistic because the most important stuff is also the hardest to test."</li>
</ul>
</li>
A useful step might be to just get tests in place on your own machine without worrying about CI initially (they were talking about training a developer, but it's a useful thought for being efficient tackling legacy projects) (9:30)</li>
"Having high quality code is almost impossible without good test coverage. Test coverage enables refactoring. If you can't refactor your code it's really hard to keep it in good shape." (17:15)</li>
As you flush out bugs by adding more and more coverage: "The type of issues you find over time, the complexity goes up, because they're not really related to the interplay of something known, they're usually related to more and more complications; instead of just one unit of code interacting... [it's] 5 or 10 or 200; these larger components and permutations that you never considered in the first place, because some customer hit it for the first time." Moments where you go "woah I never even knew that was possible" (25:50)</li>
"It all comes down to managing risk, risk of the software in question being correct" (30:20)

"There's a risk of producing incorrect software, but there's also a risk to the business that the engineers will not be able to keep pace with the business if they have to spend so much extra time just verifying the software, there's a mistake in thinking that writing tests takes more time, the reality is that if you have automated tests then you will be able to work much faster than if you didn't"</li>
</ul>
</li>
On a project that had tests once - "...at some point tests had stopped passing", should you "blow them away or fix them?". (32:22)

For an "application that's been developed for two years with no tests with this big team and they don't have the skills; that's going to be a big up front investment, it's going to be a year or two before [adding test coverage] starts to pay off."</li>
"... at some point tests had stopped passing and they were never updated - okay do I trust these or not? If they're not passing, do I blow them away or fix them." - In this case blew them away and spent three months writing tests such that he could build new features. "Even now [the project has] low test coverage. This has been a multi-year effort."</li>
"This is one of the real problems when it comes to testing, is that if for some reason the people who start the project don't do testing, and the project is successful enough to go on for X number of years without it, then the effort required to add testing after the fact is an order of magnitude than it would have been at the beginning." ... "the can gets kicked down the road."</li>
"Frustrated by the attitude of 'let's not spend time making this code high quality because what if this startup fails' but what if this startup succeeds? Yes it may be possible you only survive by cutting corners but not convinced, think it might be a delusion."

"The only way to go fast is to go well." ~ Bob Martin</li>
</ul>
</li>
</ul>
</li>
"Why is good code good? It's fast to work with. I prefer to say 'the code is understandable' because that makes it more clear. Good/bad sounds like inward looking practicing of a craft. 'Not Understandable' is more tangible - that means it's time-consuming and expensive to work with. Understandable code is faster and less expensive to work with." (35:20)</li>
</ul>

"It all comes down to managing risk"</p>
</blockquote>
The people on the episode are Ruby developers, but the lessons they talk of apply regardless of language.</p>
(Some of the quotations above are paraphrased)</p>
Thanks, now I'm even less sure what to do</h2>
Are you wrestling with something you don't like in a codebase you have to deal with? I'm guessing that's why you're here. Or maybe someone sent you this along with a rant about the tech debt in the codebase you own.</p>
Either way, I suggest slowing down a bit, sitting down together (while maintaining social distancing), and considering how all the things that bother you about the code in front of you fit in to the various taxonomies detailed above. Then use that assessment to make a calm and rational plan about roughly where you want to get to. Then decide on the one</em> next thing you will do towards that. Use the Ron Jeffries approach to get you there without wasting time on things that don't really matter.</p>
Practically, that might mean if you have a user story, ticket or Trello card for a feature, it might take longer to do as you include the work to "pay down" some of that badness along the way, knowing that it will improve your overall velocity over time. Be wary of pulling out separate "debt" stories to do, though that can work if the team dynamic is right.</p>
To keep your stress levels down make it a shared team problem, have a bit of a laugh about it, take regular breaks in the great outdoors, and support each other.</p>
Good luck!</p>
I'll leave you with this little song I found on YouTube about bugs:</p>
</iframe>
Further Reading</h2>
If you want to avoid the task at hand by reading the whole internet in the hope that it will help, start here:</p>

A lengthy set of opinions on what to do with a huge 10 year old untested pile of spaghetti code:  https://softwareengineering.stackexchange.com/questions/416242/is-it-the-correct-practice-to-keep-more-than-10-years-old-spaghetti-legacy-code</a></li>
Pluralsight have written an excellent long-form article on the subject: https://www.pluralsight.com/blog/software-development/erasing-tech-debt</a></li>
The Engineering Manager on technical debt: https://www.theengineeringmanager.com/growth/how-to-argue-the-space-to-tackle-technical-debt/</a></li>
https://adevait.com/software/technical-debt</a></li>
https://medium.com/@adamberlinskyschine/wtf-is-technical-debt-b9e9d5f89d9</a></li>
https://wtfismyengineertalkingabout.com/2017/03/18/wtf-is-technical-debt/</a></li>
https://codingsans.com/blog/technical-debt</a></li>
https://builtin.com/software-engineering-perspectives/technical-debt</a></li>
https://www.bmc.com/blogs/technical-debt-explained-the-complete-guide-to-understanding-and-dealing-with-technical-debt/</a></li>
https://dzone.com/articles/what-technical-debt-it-and-how-to-calculate-it</a></li>
https://bigrivertech.com/technical_debt_assessment/</a></li>
https://www.cutter.com/consulting/technical-debt-assessment-and-valuation</a></li>
https://martinfowler.com/tags/technical%20debt.html</a></li>
https://codeclimate.com/blog/10-point-technical-debt-assessment/</a></li>
http://thinkapps.com/blog/development/technical-debt-calculation/</a></li>
https://www.productplan.com/glossary/technical-debt/</a></li>
https://leadership.garden/tips-on-prioritizing-tech-debt</a></li>
https://www.steveonstuff.com/2022/01/27/no-such-thing-as-clean-code</a></li>
https://leadership.garden/tips-on-prioritizing-tech-debt/</a></li>
Youtube: "Types Of Technical Debt And How To Manage Them" ~ Dave Farley</a>

"Sense of design" over life of system, not just good design up front.</li>
Measure of quality is ability to make changes</li>
Metaphors for different types of tech debt:

Family loan</li>
Bank loan (lack of tests)</li>
Loan shark (visit from knuckles)</li>
Organised crime loan (fatal if not repaid. Legacy technology)</li>
</ul>
</li>
</ul>
</li>
Youtube: "7 Ways to Get Ahead of Technical Debt & Keep It Under Control" ~ Rob Walling & Derek Reimer</a>

Seen a business acquisition fail due to tech debt found during "due diligence"!</li>
</ul>
</li>
The "Lava Flow" anti-pattern where code is written, ossifies and leaves an archaeological record of all previous design attempts, libraries and technologies in the current production code rather than the git logs where they belong

https://mikehadlow.blogspot.com/2014/12/the-lava-layer-anti-pattern.html</a></li>
http://antipatterns.com/lavaflow.htm</a></li>
https://en.wikipedia.org/wiki/Lava_flow_(programming)</a></li>
</ul>
</li>
</ul>


Personal backlogs
2020-06-30T00:00:00+00:00
What's wrong with our current backlog / icebucket / todo list?</h2>
Do you have a backlog of doom? A bottomless bucket full of probably unimportant things you possibly should do, interspersed with the occasional really import thing that will bite you hard if you don't do it? Do you spend time repeatedly refining the list to make sure you don't miss the important things? Or do you just ignore it and hope luck will save you from the things you let slip?</p>
I've known for a long time that basecamp promote having no backlog at all</a>, based on the idea that it takes too much time and anything important will come up again anyway. To some extent I agree, but I have also found serious issues in systems that are perhaps less visible to the naked eye which will for sure have a consequence if not addressed (maybe something that is quietly corrupting data or failing to store it, or maybe a security issue).</p>
So what should we do instead?</h2>
The answer came to me when I read basecamp's new Shape Up book (summarized here)</a> that describes their unique way of working. They don't have a shared backlog, instead they leave it to each person in the organisation to use whatever method they choose to remember what they</em> think is important, and then bring that to the table for consideration at the scheduled times that new work is considered.</p>
The full shape-up process is a bit too much of a jump for the teams I usually work on, however I shared the above with a team I worked with, and they took it and came up with an excellent best-of-both-worlds solution. And it's so good I think everyone should use it.</p>
This solution...</p>

perfectly balances the elimination of a backlog-of-doom nightmare,</li>
addresses the need to not lose important ideas as team members change,</li>
and adds the ability to collaborate.</li>
is ridiculously simple</li>
</ul>
Trello</h3>
Set it up as follows: Set up a new trello board, name it "shared backlog" or similar, create a list (column) for every team member with their name (and optionally role) as the list title.</p>
</p>
Now when anyone thinks of anything that the team should possibly do, they add a card to their own list</strong> on the shared backlog trello board. It is now clear who owns the idea and is responsible for campaigning to get it prioritized.</p>
In preparation for the next planning session each person organises their own list so that the things they think are most important are on the top.</p>
On a regular basis, (say fortnightly during a sprint planning session or similar), the team goes through the board, and each person talks about their top one or two cards to say why they think it should be included in the sprint. The cards that are chosen by the team get added to the sprint to be done as part of the normal work for the team, thus making sure they actually happen.</p>
Simple eh?</p>
Here's an example board https://trello.com/b/cxRXPgcb/personal-backlogs-example</a>, you can use it as a template if you wish.</p>
Jira</h3>
If you're using Jira you can achieve the same effect by abusing the "sprint" feature. Just create some extra sprints with the "Create spring" button and instead of calling them "sprint 25" etc, call them "Tech backlog", "Design Backlog", "Customer Service Backlog" etc. Then you can nicely group each role's needs and they can manage their own priorities within their own backlog. The actual "backlog" in Jira can now be just for true user stories / product features.</p>
</p>
Useful?</h3>
Let me know</a> if you use this approach and if it works for you or not.</p>
What about normal feature development?</h2>
You could theoretically have the product owner manage all the forthcoming product features in their column. I think it is better for them to continue to manage the product's feature backlog in the existing jira/trello/github project system, which will now be beautifully free from tech debt, random ideas and other clutter. This is what we did and it worked great.</p>
Bonus ideas</h2>

Add a label "next sprint", and get people to tag the cards they think should happen next.</li>
Technical debt often ends up here, read this http://laughingmeme.org/2016/01/10/towards-an-understanding-of-technical-debt/</a> to be able to talk more coherently about this topic.</li>
Turn on voting in trello and vote on each other's cards.</li>
</ul>
Did it work for you?</h2>
This setup worked superbly for us as a team and I'd recommend it to anyone who'll listen.</p>
We no longer have the overhead of a product owner losing the will to live trying to understand cryptic tech debt cards left by a previous developer, wondering if they were trivial or pending catastrophes.</p>
When someone leaves, the new person or others on the team can look over their cards and decide what to move into their own column.</p>
Also available on the podcast</h2>
If you like this, you might want to have a listen to me and David talking about this and other lessons learned in episode 8 of software should be free - Hiding from Covid-19 and the end of 2 years at DfE</a>.</p>
Further reading</h2>

A very similar idea of "JIT selection from independent streams" by Jason Cohen (aka A Smart Bear)</a> which is well worth reading.</li>
</ul>


Fast backup to external drive with lz4
2020-04-23T00:00:00+00:00
The simplest backup is to just tar-gz your home directory, but it's painfully
slow. A friend</a> put me on to lz4</a> which isn't installed by default in ubuntu &
mint, it moves the bottleneck from the compressor to disk i/o for my spinning
rust usb disks.</p>
The default output of this is basically silence for two hours while it runs
which isn't great. Enter pv</code> (pipe
viewer</a>)
which can show you progress based on byte count.</p>
Install lz4 and pv with:</p>
sudo apt install liblz4-tool pv</span></span></code></pre>
And then pipe the output of tar through pv and into lz4</p>
Here's a quick script for backing up my home folder to an external drive;
customize to suit your needs:</p>
#!</span>/bin/sh</span></span>
#</span> https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh</span></span>
echo</span> "</span>Opening/creating backup folder...</span>"</span></span>
mountpoint</span>=</span>/</span>m</span>e</span>d</span>i</span>a</span>/</span>t</span>i</span>m</span>/</span>b</span>a</span>c</span>k</span>u</span>p</span>/</span></span>
cd</span> "</span>$</span>mountpoint</span>"</span></span>
mkdir</span> -</span>p</span> fox</span></span>
cd</span> fox</span></span>
base</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span></span>
src</span>=</span>t</span>i</span>m</span></span>
echo</span> "</span>Getting source folder size...</span>"</span></span>
size_bytes</span>=</span>`</span>du</span> -</span>sb</span> "</span>$</span>base</span>/</span>$</span>src</span>"</span> |</span> awk</span> '</span>{print $1}</span>'</span>`</span></span>
echo</span> "</span>$</span>size_bytes</span> bytes to backup.</span>"</span></span>
echo</span> "</span>Backing up...</span>"</span></span>
tar</span> -</span>cpC</span> $</span>base</span> $</span>src</span> -</span>P</span> |</span> pv</span> -</span>s</span> "</span>$</span>size_bytes</span>"</span> |</span> lz4</span> >></span> "</span>$(</span>date</span> -</span>d</span> "</span>today</span>"</span> +</span>"</span>%Y%m%d-%H%M</span>"</span>)</span>-home.tar.lz4</span>"</span></span>
echo</span> "</span>Done.</span>"</span></span></code></pre>
This script is also available here:
https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh</a></em></p>
Things to watch out for</h2>

Progress is inaccurate and variable because it's based on input bytes
processed vs total, but the speed is limited by output bytes to the spinning
rust backup disk, and the ratio varies with the compressability of the input
data.</li>
Finding a file will require reading or decompressing the entire archive
because of the way tar works.</li>
</ul>
Keeping it simple</h2>
There's lots of great backup tools for backing up home but most of them create
obscure custom formatted backups, often with incremental backups and chunked up
files for efficient use of space. While that's great I know from experience
that you don't always have all the right tools available to unpack such things
when you really need them, and there's always the chance that you are missing
an important incremental. In short complexity is worrying when it comes to
backup (and especially restore).</p>
References:</h2>

https://stackoverflow.com/questions/24063846/how-to-use-tar-with-lz4#24086155</a></li>
https://superuser.com/questions/168749/is-there-a-way-to-see-any-tar-progress-per-file/665181#665181</a></li>
https://stackoverflow.com/questions/8228047/adding-timestamp-to-a-filename-with-mv-in-bash/8229220#8229220</a></li>
</ul>


Question: where are we at with dynamic static sites?
2020-04-14T00:00:00+00:00
So tell me about dynamic static sites...</p>
History lesson</h1>
I've been a software engineer primarily creating dynamic websites for a loooong
time. First there was a server with html files and caching proxies. Then there
were scripting extensions (cgi, php, asp-classic etc) to spice up your html,
then there were full mature web frameworks for all the things (asp.net mvc,
ruby on rails etc).</p>
But then we got traffic and libraries and platforms and sdks and databases in
the cloud and everything became slow. 😞</p>
So we added CDNs to make it all fast, and cache invalidation to make the
engineers cry.</p>
Now what's old is new again, but with a twist, static sites pre-generated by
dynamic code (such as my blog created with jekyll). We get the benefits of old</p>

CDNs love static sites; caching and proxying work again; and cache
invalidation still makes us cry.</li>
</ul>
The new saviour is serverless (functions as a service) so that we can still add
dynamic stuff to these static sites.</p>
Yes I can go read endless articles about how to do</em> all this stuff, and turn
my CV and skills on their head so I can sell the hot new thing. But what I
actually want to know now is....</p>
What are you</em> doing with dynamic static sites?</h1>
Have you tried it? As a pet project? As a commercial project?</p>
Did it go well? Is it the new future or is it just for niche needs?</p>
I feel like because this is all sooo</em> new, that it's not clear yet how well
adopted this stuff is or will be, and how much of the old is being thrown out
and rewritten.</p>

My opinion...</h1>
...based on good knowledge of tech but not based on any actual experience is that
it depends entirely on the type of thing you are building:</p>

Low volume intranet/extranet style sites - don't bother, asp.net / rails is
fast enough, and it will cost you more to develop a blended static/dynamic
thing.</li>
E-commerce: totally worth it because you can handle any spike in traffic to
your front page, and serverless lets you seamlessly scale the dynamic parts.
Caveat it will be more expensive to build</strong> but that's okay because money in
is proportional to how many customers you can serve well.</li>
Government or other highly accessible services - don't do it. You can't meet
the accessibility needs if half your site's critical behaviour is done in
client-side javascript with calls off to serverless functions in the cloud. I
might be wrong about this, I hope I am. Maybe it's possible but just hard.</li>
Your blog and other traditionally CMS driven sites: just use jekyll for your
own blog</a> and put it on github pages.
For more complicated commercial sites static site generation is a big win in
terms of technical complexity reduction, reliability improvements and speed
improvements for users. Which static site generator you use will depend on your
organisation's internal needs. (Can everyone use git? Or do you need a UI for
people to type in? How much money to you have?)</li>
Etc. (there are sooo many reasons for tech, I've just picked a few)</li>
</ul>

Speak out now</h1>
Before you go, comment below.</p>
What have your experiences been with the shift to dynamic-static sites? Have
you done any of it? Do you think I'm right/wrong about any of this?</p>


What you should do between contracts
2020-04-09T00:00:00+00:00
Strange times with Covid-19</h1>
I can't post this without mentioning the context I'm writing in of the
coronavirus pandemic lock-down.</p>
I consider myself extremely lucky to be in the line of work I am that allows me
the flexibility to continue working remotely whilst still protecting myself and
those around me by avoiding contact.</p>
What I normally do</h2>
In calmer times I'd have taken the opportunity to spend at least a couple of
weeks with family (have small people and time with them is preciously
fleeting). Then get my life in order, then go hell for leather getting into the
next thing I can help build.</p>
Some of my fellow contractors like to make sure they have the next contract
lined up ready for the finish date of their current contract. While this is
optimal for revenue I don't do this because:</p>

I find it is a full time job generating, tracking and dealing with
contracting leads</li>
I wouldn't want to be distracted from the current client's work to the next
thing</li>
I wouldn't want to half-arse the contract hunt and not get the optimal
contract for myself and the next client.</li>
There is a tendency for last-minute contract extensions to appear which would
mean letting down one or other client.</li>
</ul>
So after a contract is completely finished, and some time with family, only
then do I take the contract hunt seriously.</p>
What I'm doing this time</h2>
Given the uncertainty caused by COVID-19 + IR35 (by the way COVID is short for
Corona Virus Disease) and the fact I just moved house (i.e. less cash reserves
than usual) I can't take it too easy this time. Even though the IR35 changes
have been delayed a year a lot of the damage has been done so the contractor
market is challenging at the moment.</p>
I finished my contract with DfE on a Tuesday, allowed myself till the following
Monday for uninterrupted family time. (Okay almost uninterrupted, I can't
really put technology down for that long and I also have my charitable work for
DogLost</a> that takes up some time.)</p>
Now that is over, I'm doing roughly 9am-1pm on the business related tasks
suggested below.</p>
What should you do to make the most of finishing a contract?</h1>
Step 1. Take a break</h2>
Contracting is in my experience much more intense than permanent employment.
This is in fact how I like it. But in order to be able to give your all to the
next contract I think it is important to give yourself the space to recharge.</p>
When in a contract it's easy to end up with tunnel vision, especially for one
as long as my last one (2 years!), where all you can think about is how it was
done there and how you reacted to the influences around you during that time.
One of the strengths you can bring to a contract as a contractor is the broader
perspective from seeing many organisations, approaches, technologies and
people. It requires a bit of down-time for your latest experience to sink in
once you are out of the hustle and bustle of delivery, and for it to be merged
into your bank of knowledge.</p>
What better way to end a contract than with good chunk of time with family and
friends while your subconsious churns through and processes everything that's
happened over the last contract.</p>
I believe if you dive headlong into the next thing with nothing more than a
normal weekend off then your brain will not have the opportunity to properly
process what you've learnt before being overwhelmed with an influx of new
information from a new client.</p>
Step 2. Update your online presence and CV</h2>
You've probably learned a lot and changed a bit since you started the last
contract.</p>
Review and update all your profiles:</p>

LinkedIn</li>
Twitter (make a new pinned tweet!)</li>
GitHub</li>
StackOverflow</li>
etc.</li>
</ul>
Think about what your next client will be interested in and what they want help
with and make sure your profiles provide evidence that you have done similar
things before and can get results.</p>
Step 3. Do some writing</h2>
The best way to shape your thoughts on the whole thing is to put them into
writing. So spend some time blogging like I'm doing now, and make sure to
cross-post your articles to the places where your customers will see them. You
should own your own content, keep it on your own domain on your own blog where
no platform can take away your audience, then cross-post to places like
dev.to</a>, LinkedIn and Medium (if that's your thing) with links
back to your own domain. Better still on your posts ask people to sign up to
your mailing list and email them when you post new articles etc.</p>
Step 4. Catch up with old acquaintances</h2>
It's hard when you're flat out with life and contracts to keep in touch with
everyone. Reach out to old business and personal friends. You never know it
might kick off your next opportunity, or help you learn something about
yourself that's useful.</p>
Step 5. Go all guns blazing on getting the next piece of work</h2>

"You will get all you want in life if you help enough other people get what they want,"
~ Zig Ziglar</p>
</blockquote>
Figure out who'd value your skills most even in these difficult times and have
at it.</p>
You might find my previous article "How to find contract developer
jobs"</a> useful
at this point.</p>
Personally I'm looking to also increase the value I provide beyond
implementation work. This might take the form of team lead roles, or some
digital transformation consultancy.</p>
~ End ~</h1>
What do you do between contracts or how do you avoid having gaps? Are you
between contracts now thanks to IR35?</p>


Time to "Shape Up" your SCRUM processes? The new thing from Basecamp
2019-11-26T00:00:00+00:00
Basecamp, who are famous for carving out their own path in the software world
have documented and shared their own way of defining and building their
software-as-a-serice (SaaS) product, also known as "basecamp". You can read the
whole thing here: https://basecamp.com/shapeup</a>.</p>
If like me you've mostly worked with SCRUM/Agile/Kanban style teams you should
pay attention to the ideas they share. Unlike many people who teach and follow
the wikipedia truth of SCRUM as the</em> way of building software, basecamp have
not satisfied themselves with cargo-culting "industry best practice" and have
instead carefully honed their own particular way of getting the best out of
their resources.</p>
Although the "ShapeUp" process is finely tuned to suit basecamp, there are many
things in there that the average SCRUM team could take inspiration and
improvement from without throwing out everything in place at the moment. An
example I particularly like is the way everyone can bring a pitch to the
betting table, from their own personal backlog of priorities, contrasted with
the usual approach of throwing yet another another card/story onto the
team-shared dumpster-fire of a backlog / todo-list and hoping it somehow
magically gets to the top of the list just before all your chickens come home
to roost.</p>

Key features of shaping up</h1>
Shaping</h2>

An iterative process of taking concepts from initial idea to something that
could be built with bounded risk and expectations.</li>
A largely solo or pair exercise, but done in consultation with experts on the
wider team.</li>
</ul>
Pitch</h2>

The output of shaping.</li>
A detailed but high-level document outlining what should and shouldn't be
built, with room for discretion and design work by the designers and
programmers.</li>
</ul>
Bets</h2>

There is a meeting where pitches are discussed, prioritised and scheduled for
the next cycle of work, they "take a bet" on a pitch.</li>
If a pitch blows its timebox, then the default is always to stop and it
doesn't get any special priority in the next cycle. This limits the risk of
runaway unstoppable projects.</li>
</ul>
6-week cycle with gaps</h2>
Basecamp operate a 6-week period of building what was decided in shaping and
betting. This can either be one 6 week bet or set of 2 week bets.</p>
Once this is done there is a 2-week gap where people can do maintenance, tech
debt paydown, plan new things etc.</p>

If you'd like to hear a bit more about ShapeUp, have a listen to
https://pod.timwise.co.uk/5</a> where I go over their process in a bit more
detail with regular co-host David.</p>
I've also written up a nice approach to avoiding a "backlog of doom" in my
personal backlogs post</a>.</p>


Should you rebase or merge to update feature branches in git?
2019-10-14T00:00:00+00:00
You have a "feature branch" in git that you've been working on for a while but
now main</code> or master</code> has moved on.  You know of merge</code> and rebase</code>, but
which one should you use? And what can you do to avoid being in this position
in the first-place?</p>
TLDR</h2>
Try rebase. If that dissolves into conflict-resolution-hell then give up, merge
master into your branch and move on.</p>
The Options</h2>
You need to bring your feature branch up to date with with master to flush
out any incompatibilities and deal with any merge conflicts.</p>
You have two common choices:</p>

Merge</strong> origin/master</code> into your branch.</li>
Rebase</strong> your branch onto origin/master</code> and force-push.</li>
</ul>
The Trade-offs</h2>
A blanket rule here either for merge or rebase is unhelpful because there are
trade-offs</strong> to be made that vary depending on the specific circumstances. (Isn't
that always the answer with git?! "It depends!")</p>
Should You Merge?</h3>
A merge from master</code> is done like this:</p>
git fetch</span></span>
git merge origin/master</span></span>
git push</span></span></code></pre>Merge - The Good</h4>

👍 Reliable no-brainer that anyone can follow by rote.</li>
👍 Resolve conflicts only once.</li>
👍 Accurate representation of what happened over time.</li>
👍 Avoids retrospectively introducing bugs and test failures into commits that
used to be
valid</a>.</li>
👍 Avoids re-writing previously shared branch, which can confuse less
experienced git users if they are working with you on the branch.</li>
</ul>
Merge - The Bad</h4>

👎 Doing this repeatedly makes for a messy history for little or no benefit.</li>
👎 Significant merges from master makes it harder/impossible to then go back and
clean your branch's commits with a git rebase --interactive</code>.</li>
👎 Tends to generate wide tramlines in the commit
history</a>
that can be very hard to follow  when looking back to find out when/why
something was done. (mitigated by git log --first-parent</code>, until
you need to dig into a branch).</li>
</ul>
Should You Rebase?</h3>
A rebase onto master is done like this:</p>
git fetch</span></span>
git rebase origin/master</span></span>
git push --force-with-lease</span></span></code></pre>Rebase - The Good</h4>

👍 Avoids tramlines generated by long-lived feature branches
branch</li>
👍 Makes resultant history in master</code> much easier to follow</li>
👍 Reflects the intention more clearly of "merge these commits into master" as
opposed to "here's how I flailed my way to a working thing"</li>
</ul>
Rebase - The Bad</h4>

👎 Can confuse less experienced git users if they are working with you on the
branch (the answer is usually for them to run git pull --rebase</code>)</li>
👎 Results in resolving conflicts multiple times (screencast)</a></li>
👎 Loses chronological order of creation of code (personally I think this is
less important than a series of clean intentional patches to be applied to
the codebase when merged to master</code>)</li>
👎 Could in somewhat rare circumstances retrospectively introduce bugs and test
failures into commits that used to be
valid</a></li>
</ul>
Heuristics To Use</h2>
Try rebase. If that dissolves into conflict-resolution-hell then give up, merge
master into your branch and move on.</p>

"Try rebase. If that dissolves into conflict-resolution-hell then give up,
merge master into your branch and move on."</p>
~ Tim Abell</p>
Tweet this</a></p>
</blockquote>
Rebase is my preferred approach until:</p>

Rebase becomes too costly to fix up due to conflicts with master</code>, or</li>
I become aware of an incompatibility with master</code> that changes the meaning
of the previous commits and needs serious work to resolve.</li>
</ul>
You can</em> usually make a difficult rebase work, and I've hunkered down and
tackled probably more than I should have in the name of perfect history graphs.</p>
The problem with a tricky rebase is that if you are doing this for business and
not just for fun then there is a major time cost for only a marginal benefit.</p>
How to make it through rebase conflicts unscathed</h2>
If you decide to battle on with rebase in-spite of conflicts then my tip for
you is:</p>
Don't jump straight to the "correct" code when fixing each commit's conflict,
as that guarantees the next commit won't apply.</p>
Instead as you work through the rebase make each commit apply with its original
meaning and nothing more.</p>
It's worth remembering that each commit on your branch describes how to change
the source code from a before-state to an after-state; so if you change the
after-state of one patch, then the next patch will no longer apply.</p>
How to avoid the pain of rebases and merges entirely</h2>
Pain around this topic is likely a symptom of not breaking down your stories /
pull requests / features into small enough chunks. On a fast moving team
master</code> is very fluid and any large & long-running branches will be hard to
review and merge. Try to chip off smaller increments and ship those, maybe
using feature flags or hidden features (those with no visible way of getting to
them).</p>
More Resources</h2>
In general merge vs rebase generates much debate, such as that found on
stackoverflow:
https://stackoverflow.com/questions/804115/when-do-you-use-git-rebase-instead-of-git-merge</a>
but it is often lacking context.</p>
There are many other articles on the merge/rebase topic such as
https://derekgourlay.com/blog/git-when-to-merge-vs-when-to-rebase/</a> but I
couldn't see anything that matched my heuristic for tackling feature branch
updates so I wrote this one.</p>
"Semantic merge conflicts</a>" are where git reports no conflict but nonetheless the code is broken.</p>
I also wrote "GitHub rebase and squash considered harmful</a>" which address a github specific horror.</p>
Get in touch</h2>
Hey there! Thanks for reading!</p>
This post gets far more traffic than anything else on my blog. I'd love to know what brought you here and if the above was helpful.</p>
Please take a moment to fire an email to me at tim@timwise.co.uk</a> and tell me a bit about yourself.</p>


Why I made SQL Schema Explorer open source
2019-10-08T00:00:00+00:00
What is SQL Schema Explorer?</h1>
It's a database schema & data browser that can follow foreign keys and draw
diagrams that I spent years building because I wanted it to exist and I thought
it might make a good first product.</p>
Why charge?</h1>
I'm pretty busy these days but I like working on software and building things
like this, to justify the amount of time that goes in I can no longer just have
it as a time-consuming hobby. I've also been learning about entrepreneurship as
a possible extension or next step from time-for-money contract-programming
services.</p>
The idea was that if I could get enough yearly subscribers to the software then
I could get balsamiq-like success and focus on making it better and better for
the people using it.</p>
Why open source now?</h1>
I got plenty of interest and interactions by my standards, more than I've ever
had for something I've published; but I never got more than one good friend paid
up beyond a generous trial. The conversations I had with people were full of
enthusiasm for the idea but not for parting with money to buy it. One
particular conversation was very positive until the idea of putting the
theoretical purchase in front of the manager was raised and then suddenly it
didn't seem viable at all; which I think speaks volumes. Useful, but not useful
enough to risk any political capital to get it, even for a pretty insignificant
amount of money.  I then had a coaching call with Justin
Jackson</a> and as part of that he suggested putting it
to one side. Up to this point I'd been unable to leave the idea alone; it had
been something I really wanted to exist. I've always worked with relational
databases, and this seemed like a painful gap. At this point I had finally
added every feature that I'd ever dreamed it should have, and the intrinsic
motivation to just build for the sake of building was waning, and without the
demands from paying users for more features it's hard to justify continuing to
pour hours in even if it is on the train.</p>
At this point there were 100 people on the mailing list (drip sent my first
bill!), one paying user/fan (Hello David! Many thanks!), and zero MRR (monthly
recurring revenue). I'd spent a bit of money on hosting (I'm still paying
digital ocean to keep the demo site</a> up).</p>
I didn't want to let down the few people who had really liked it by having it
just vanish into the ether like so many failed startups. Fortunately this is
downloadable and installable software, not a SaaS that would have to be shut
down. I've always liked the open-source ethos, and given this was no longer a
direct financial opportunity it made sense to me to give it away freely under a
copyleft license.</p>
It feels like having this as a portfolio piece given I sell coding for a living
won't do me any harm, and it makes it easier to take with me to client
projects, where I can use it to help them improve their database.</p>
Schema Explorer for client work</h1>
Here's some things I've been using schema explorer for to help my contracting clients</p>

Finding missing foreign keys without looking up complex pg_schema</code> queries</li>
Analysing data in tables and sharing screenshots with the teams</li>
Sharing screenshots of the generated diagrams to improve team understanding</li>
Using it as a tool for discussing database design</li>
Finding records needed for troubleshooting and debugging</li>
Improving my mental model of the database structure</li>
</ul>
Tell me more</h1>
It's now A-GPL</a>
which requires sharing any modified source code even if you are just letting
users access the generated site. If I'm giving my years of hard work for free,
I think it's only fair to require people to contribute their improvements back
to the community.</p>
You can find the source code at https://github.com/timabell/schema-explorer</a></p>
Feedback</h1>
Please do get in touch if this sparked a thought or interest. Just a simple
email will do tim@timwise.co.uk</a></p>
Thanks for listening.</p>


Laptop setup notes
2019-08-20T00:00:00+00:00
This is mostly for my benefit YMMV. Used for XPS13 and XPS15. Last run with Linux Mint Cinnamon 22</p>
Things to go from blank machine to fully functioning work laptop.</p>
Why not use machine images?</h2>
Because when you want to upgrade your OS image you have to do this anyway.</p>
But this will break!</h2>
Indeed, and it does, a continuous labour of love. Ideas welcome!</p>
OS install</h2>

Download mint cinnamon x64 torrent</a></li>
Burn to USB stick</li>
Configure Dell XPS 13 to use AHCI mode</a> (one time fix)</li>
Boot to stick with F12</code></li>
Install mint

Full disk encryption (LUKS), default whole disk partitioning</li>
</ul>
</li>
Run updates and reboot sudo apt update && sudo apt upgrade</code></li>
</ul>
Hibernate / hybrid sleep (unfinished)</h2>
Not working at moment, needs more research.</p>

Enable hibernate?

Bigger swap?</li>
</ul>
</li>
</ul>
Unfinished research into missing hibernation</h3>

https://superuser.com/questions/1539378/what-size-swap-partition-will-allow-a-mint-19-3-installation-with-64g-ram-to-hib?noredirect=1&lq=1</a></li>
https://superuser.com/questions/1434301/enabling-hibernate-on-linux-mint-19-1?noredirect=1&lq=1</a></li>
https://help.ubuntu.com/community/SwapFaq</a></li>
https://unix.stackexchange.com/questions/568093/enable-hibernation-in-power-management</a></li>
https://dewaka.com/blog/2021/04/08/linux-mint-hibernate/</a></li>
https://www.fosslinux.com/45454/enable-hibernate-mode-linux-mint.htm</a></li>
https://askubuntu.com/questions/12383/how-to-go-automatically-from-suspend-into-hibernate</a></li>
https://www.reddit.com/r/linuxmint/comments/93ta9u/enable_hibernation_in_linux_mint_19_tara/</a></li>
</ul>
Packaged software install</h2>

Bootstrap my dotmatrix package list my bootstrap file</a> like this (don't laugh/cringe):</li>
</ul>
curl https://raw.githubusercontent.com/timabell/dotmatrix/main/software/bootstrap.sh | sh</span></span></code></pre>
This installs my usual package list</a></p>
Hardware</h2>
Video</h3>

Disable nvidia GPU in favour of intel card</li>
</ul>
sudo prime-select intel</span></span></code></pre>Bluetooth</h3>
Enable fast-connect etc for bluetooth keyboard/mouse</a>.</p>
zsh</h2>

Switch to z-shell chsh -s /usr/bin/zsh</code></li>
</ul>
Ref: https://askubuntu.com/questions/131823/how-to-make-zsh-the-default-shell/131838#131838</a></p>
inotify</h2>

inotify increase for:

RubyMine</li>
Guard - https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers</a></li>
Syncthing - https://docs.syncthing.net/users/faq.html#inotify-limits</a></li>
</ul>
</li>
</ul>
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p</span></span></code></pre>File sync with syncthing</h2>

Run syncthing-gtk from system menu (auto-installs syncthing binary)

Configure syncthing</a> not to route over internet
(no relay, no public discovery, no NAT traversal)</li>
</ul>
</li>
Configure firewall-config</a>
to allow syncthing

Add manual service syncthing</li>
Allow tcp 22000 and 21027 udp</li>
Make home network use home zone</li>
Add syncthing to home zone</li>
Runtime to permanent</li>
</ul>
</li>
</ul>
</p>

Remove old install machine name from syncthing network</li>
Tell new and old devices about each other in syncthing</li>
watch ~/Documents</code> etc. magically sync (amazing)</li>
</ul>
All the things from dotmatrix</h2>
cd repo/dotmatrix/</span></span>
bin/symlinks.sh</span></span>
bin/install</span></span>
software/delta.sh</span></span>
# etc</span></span></code></pre>Link local git config</h2>
cd ~</span></span>
ln -s ~/Documents/config/.gitconfig.local</span></span></code></pre>Desktop/system config</h2>

Setup touchpad

Set mouse and trackpad speed/acceleration to max</li>
Enable horizontal touchpad scroll</li>
</ul>
</li>
</ul>
</p>

Setup terminal colours and turn off transparency</li>
</ul>
</p>

Turn off all the system sounds and turn the volume down to 0%.</li>
Customize time in task bar to  📅 %a %e %b %Y   🕓 %H:%M:%S</code>.</li>
Lock screen with Win+L</code> - start > keyboard > system > lock screen (Windows user habits).</li>
Turn on automatic updates in update manager preferences, and automatic obsolete kernal removal.</li>
Configure nemo file manager to always use list view.</li>
Add workspace switcher applet to taskbar.</li>
</ul>
Firefox</h2>


Sign-in to sync</p>
</li>

Remove firefox tabs (because I use Tree Style Tabs</a>)</p>

cd ~/.mozilla/firefox/<profile_id_folder>/</code></li>
mkdir chrome</code></li>
echo '#TabsToolbar { visibility: collapse !important; }' >> chrome/userChrome.css</code></li>
about:config > toolkit.legacyUserProfileCustomizations.stylesheets</code> > true</code></li>
restart firefox</li>
</ul>
</li>
</ul>
Reference: https://superuser.com/questions/1268732/how-to-hide-tab-bar-tabstrip-in-firefox-57-quantum#1268734</a></p>
Startup list</h2>

Menu > Startup Applications</li>
Adjust to current tastes</li>
Add additional startup apps:

Syncthing GTK</li>
</ul>
</li>
</ul>
Non-apt program installations</h2>

Use Applets config to download and add Pomodoro timer</a> in task bar.</li>
Install jetbrains toolbox</a></li>
Install flatpak apps from dotmatrix</a>, sign in to them all

Give flatpak access wider home folder access</a> (for sending pics) with flatseal</a></li>
</ul>
</li>
Install all the other things in dotmatrix/software</a></li>
</ul>
SSH</h2>

Generate ssh key ssh-keygen</code></li>
Add public key to github</a></li>
</ul>
Postgres</h2>
Use a docker image</a></p>
XPS firmware updates</h2>
fwupdmgr get-devices</span></span>
fwupdmgr get-updates</span></span>
fwupdmgr update</span></span></code></pre>
refs:</p>

https://fwupd.org/lvfs/docs/users</a></li>
</ul>
Shutter</h2>

Change default save location to ~/tmp/shutter</code> (new folder)</li>
change filename to shutter_%Y%-m-%d_%NN</code></li>
</ul>
Thunderbird</h2>

Restore recent backup (from data sync)</li>
</ul>
asdf</h2>

install asdf version manager</a> for all the things (ruby, node, golang etc.)</li>
</ul>
NodeJs setup</h2>
asdf plugin-add nodejs</span></span>
bash ~/.asdf/plugins/nodejs/bin/import-release-team-keyring</span></span>
asdf install nodejs 8.16.1</span></span>
node -v</span></span></code></pre>
Keyring because nodejs packages are signed and validated
https://github.com/asdf-vm/asdf-nodejs#install</a></p>
Ruby</h2>
asdf plugin add ruby</span></span>
cd some-project</span></span>
asdf install</span></span>
gem install bundler</span></span></code></pre>See also</h2>

https://github.com/thoughtbot/laptop</a></li>
</ul>


Why every team needs a Delivery Manager (DM)
2019-07-08T00:00:00+00:00
I was explaining to someone the team structure I'm working in at the Department
for Education's DfE Digital</a> and realised not
everyone has seen this structure at work and the magic it brings.</p>
Team structures</h2>
GDS (government) team structure</h3>
In short the team looks something like this:</p>

Product Manager</li>
Delivery Manager</li>
Developer(s)</li>
User Researcher(s)</li>
Content Designer(s)</li>
Interaction Designer(s)</li>
</ul>
At the top of the whole thing there's a "Service Owner".</p>
Involved from the very early stages, but a bit less in the daily grind of building technical solutions, you'll find a "Service Designer".</p>
It's a very flat structure. It's a group of peers who all bring their own
strengths to the team, but with no hard boundaries, we all muck in with
everything now and then. I particularly like helping with user research as it
keeps me connected with the users. I'm constantly feeling guilty for not
attending enough user research!</p>
This structure is modelled on the structure of a team at the Government
Digital Service
</a>(GDS)</p>
Aside: Product who?</h4>
You may also not be familiar with Product Manager (I've not seen it as much in
private industry), so as an aside let me explain:</p>
The Product Manager is more widely known in industry as Product Owner</del> (well
that's what I thought, actually it turns out product owner and product manager
are not quite the
same</a> and there is depth to the product manager/owner question</a>).
This is someone who figures out the balance of features in a service and the
software that supports it, based on all the various constraints and inputs
(e.g. user need, security, stability, internal need). One output of this role
is prioritization of work.</p>
Private company team structure</h3>
The kind of structure I've more commonly see in the private sector is something like this:</p>

Project Manager</li>
Business Analyst (BA)</li>
SCRUM Master (in title if not in deed)</li>
Developer(s)</li>
Testers(s) / QA (Quality Assurance)</li>
</ul>
Which is better?</h3>
I've seen good and not so good people in all of the roles, and number of roles
varies with team size.</p>
I think the GDS-style team structure is superior when it comes to shipping
user-value as a development team.</p>
DM for the win</h2>
The role I want to focus on here (though they are all awesome) is Delivery
Manager as it's the most uncommon in my experience.</p>
I've personally never seen delivery management called out as a specific named
role outside of my work for "Digital" teams in UK Government. I hope this role
becomes more common everywhere.</p>
So what does a Delivery Manager actually do?</p>
Greasing the enterprise wheels</h3>
In any organisation beyond about 20 people the bureaucracy starts to sneak in,
layers of rules are added as a business tries to avoid repeating previous
mistakes, and as you hit enterprise scale it's positively stifling, to the
point that people issues and an inability to even open Microsoft Word (TM) (R)
((c) 1908)</em> on a work computer without signing 3 forms in blood</strong>, become
nearly impossible. The thing is, a dev team is usually building something that
hasn't been built before, maybe as part of a major change to how the
organisation functions</strong>.</p>
Web development is more than just taking your sales
brochures and putting them online, if that was all we wanted I'd give you a
copy of WordPress and find something else to build.</p>
Sometimes you can hide the team away in a
skunkworks</a> project, but that can't
last forever and eventually the old meets the new, and suddenly the whole
project is at risk of being rejected by the legacy organisation.  The
productivity can be sucked out of a development team for good, they can end up
unable to deliver anything.</p>
This is where a Delivery Manager</strong> role is golden; they deal with making sure
the organisation's goals to have this new service / system / product / software
are able to be progressed, conversations & endless meetings had, finance
obtained, hardware procured, forms filled, security audits managed sensibly,
pile-on demands defended … the list is endless. By having this role the team
can focus on building and shipping, comfortable that they won't be sidetracked
or blocked.</p>
I still like the way the wonderful Mark 'Stanners' Stanley put it back in 2012.
I think this cuts to best thing about it:</p>

The delivery manager is there to remove any and all things that are hindering
or ‘blocking’ them, so the team can deliver the product.</p>
~ Mark Stanley -
https://gds.blog.gov.uk/2012/12/12/a-day-in-the-life-of-a-delivery-manager/</a></p>
</blockquote>
The approach is not new, back in the year 2000 Joel Spolsky wrote:</p>

Compare this to Microsoft, where things are done at the lowest level, and
most managers act like their most important job is to run around the room,
moving the furniture out of the way, so people can concentrate on their work.</p>
~ Joel Spolsky -
https://www.joelonsoftware.com/2000/03/19/two-stories/</a></p>
</blockquote>
Running great delivery teams</h3>
You'll also find the Delivery Manager doing inward-looking work similar to the
role of "SCRUM Master". This includes setting up the right agile environment
(which may SCRUM, Kanban, or something else), tackling any dysfunction and poor
team performance that has built up, keeping in touch with all the team to make
sure they are happy and productive, running team retrospective sessions, and
sometimes hiring-and-firing to get the best team in place.</p>
Is it good for devs?</h3>
As a developer on an Agile team working inside a government team (not for the
first time), I am eternally grateful for all the things I don't</em> have to worry
about because we have a DM. It's not that I'm not capable, I'll happily turn my
hand to the biggest problem of the day for a team at any time, but the code
won't write itself and that's what I was hired to be a pro at. Having a DM lets
me focus more creating software that actually does a job for our users, and
makes sure we can do what's needed to make great systems even if that reaches
beyond the team's original boundaries.</p>
Further reading</h2>

https://www.gov.uk/service-manual/the-team/what-each-role-does-in-service-team</a></li>
https://bizfluent.com/info-8704880-role-software-delivery-manager.html</a></li>
http://itsadeliverything.com/delivery-manager-a-new-role-for-an-agile-world</a></li>
https://it.toolbox.com/blogs/deliverydr/what-is-delivery-management-070606</a></li>
https://www.greatsampleresume.com/job-responsibilities/delivery-manager-responsibilities/</a></li>
https://work.chron.com/job-description-service-delivery-manager-24247.html</a></li>
https://www.quora.com/Whats-the-difference-between-a-delivery-manager-and-a-project-manager</a></li>
https://melissaperri.com/blog/2017/06/29/product-manager-vs-product-owner</a></li>
</ul>


How to find contract developer clients
2019-06-26T00:00:00+00:00
Handling recruiters</h2>
Before you dive in, you might want to look at my page on dealing with
recruiters</a>, especially if you are going to be dealing with
multiple recruiters at once.</p>
Rate research with ITJobsWatch.co.uk</h2>
IT JobsWatch</a> is a great guide to market prices. You can see numbers, graphs, trends, compare tech, and see what appears together.</p>
It's also a useful source of ideas for considering what might be a good investment in saleable skills.</p>
Note the volume as well as the price, for example C# is massively higher volume than Ruby.</p>
Lead sources</h2>
In rough order of the success I've had with them over the 7 years of contracting:</p>
JobServe</h3>
https://www.jobserve.com/gb/en/Job-Search/</a></p>
By far the most reliable source of leads. Slightly shonky website, bit fiddly
to use but it seems that all the recruiters post their wares here.</p>
Tactics:</p>

Upload a pdf cv</li>
Run a not particularly tight search (e.g. asp.net as a keyword with 50 mile radius, ordered by distance)</li>
Blindly send your cv to all of them without worrying too much or bothering to address the post.</li>
</ul>
</p>
This might sound lazy, but in my experience I almost never hear back about the
advertised role, but what this does do is let recruiters that specialize in
your particular skills know you are actively looking, which is a lot easier for
them to make use of than hoping to tempt someone who isn't.</p>
Workable</h3>
https://jobs.workable.com/</a></p>
LinkedIn</h3>
By posting regularly the progress of my search and asking contacts for help I
have had some luck getting referrals. I've never used LinkedIn's proper job
posting system but it's probably worth checking out.</p>
It's worth being careful if you are looking before leaving. I tend to not look
till I've finished the previous contract and had some time to refresh so that
hasn't been an issue for me.</p>
LinkedIn job hunt status</h4>
You can now set your "open to work" status to everyone, recruiters etc so be
sure to set that. I've definitely had inbound interest from that, albeit
relatively low quality. It also adds an overlay to your profile image which your
network might see when you post things.</p>
LinkedIn Hiring groups</h4>

Outside IR35 contracts board</a> group. - I can do invites if you want them. Let me know.</li>
</ul>
Recruiters</h3>
It's probably worth reaching out to recruiters directly.</p>
They cold email us so no reason not to go the other way when you need a job.
Could work out for everyone.</p>
Here's a useful list that you could use as a source of leads as well as a way
of filtering all the inbound email into a folder
https://github.com/alexmbird/uk-it-recruiter-domains/blob/master/domains.txt</a></p>
People you know (your "network")</h3>
Possibly with the assistance of LinkedIn, WhatsApp, Telegram, Signal, Facebork etc.</p>
I don't like to mention this as the idea of "networking" always made me cringe,
but the truth is who you know matters and can get you an "in" where you might
not have before, and indeed in the past I've won contracts through friends of
friends and repeat business.</p>
It doesn't seem very actionable, and when you need it it's probably too late to
build a "network" of trust, but you can still reach out to the people you do
know with a friendly message and make it clear you're looking (and be clear
about what you're looking for - to trigger what Jonathan Stark calls "Rolodex
Moments"</a>)</p>
This is also a reminder to leave a trail of positive experiences with everyone you meet, go the extra mile, and invest a little regularly in your relationships whether you "need" them in the moment or not.</p>
Others</h3>
These are ones I've come across that you might want to cover if you are looking
to not miss anything, but I haven't either used or succeeded with these myself.
Suggestions for additions to this list welcome, just drop me an email or raise
a PR</a>.</p>

https://gun.io/find-work/</a></li>
https://news.ycombinator.com/submitted?id=whoishiring</a></li>
https://outsideir35roles.com/contract-listings/</a></li>
https://remoteok.io/</a></li>
https://snap.hr/</a></li>
https://uk.indeed.com</a></li>
https://wellpaid.io/</a></li>
https://www.amazon.jobs/</a></li>
https://www.contractoruk.com/it_contract_jobs</a></li>
https://www.contractspy.co.uk/</a></li>
https://www.cv-library.co.uk</a></li>
https://www.cwjobs.co.uk</a></li>
https://www.icontract.co.uk/</a></li>
https://www.itcontractjobs.co.uk</a></li>
https://www.itjobboard.net</a></li>
https://www.jobserve.com</a></li>
https://www.jobsite.co.uk</a></li>
https://www.linkedin.com/jobs</a></li>
https://www.monster.co.uk</a></li>
https://www.reed.co.uk</a></li>
https://www.technojobs.co.uk</a></li>
https://www.upwork.com/</a> - seems to be low paid commodity labour on an
international market largely, however I think some people have managed to
break through this and become in demand. At that point you're basically an
entrepreneur.

https://www.quora.com/What-should-I-do-to-increase-my-earnings-on-Upwork-when-I-have-been-registered-on-it-for-over-a-year-and-have-done-only-one-10-job-for-which-I-received-a-five-star-rating</a></li>
</ul>
</li>
https://www.yunojuno.com/</a></li>
</ul>
Deceased sites</h4>

https://jobs.github.com/</a></del> - github jobs decommissioned</a></li>
Sadly stackoverflow jobs has been decommissioned</a>. https://stackoverflow.com/jobs?j=contract</a></del></li>
</ul>
Places to find even more</h3>

https://www.quora.com/What-are-the-best-job-boards-for-software-engineers</a></li>
</ul>
Practical tips</h2>

Track everything you've applied to (Trello, or HubSpot if you're feeling rich).</li>
Keep a copy of the full advert description and rate, they are often taken down before you speak to anyone which weakens your negotiation position.</li>
Follow up with phone calls after applying - makes you more real, makes it easier for them.</li>
Track all the businesses in your sector and technology, and their hiring managers and contact them before letting recruiters get in the middle.</li>
See also my recruiters page</a> for defense against the dark arts. Or you could just blanket declare "no recruiters".</li>
</ul>
About this post</h2>
Someone I know is looking for work (hello!) so rather than just emailing I
thought I'd make a more permanent list of all the places I've come across for
finding work as a contract developer.</p>
For context, I'm a backend developer, specialising in Microsoft ASP.NET / C# &
SQL Server databases, with a smattering of other things (t-shaped people as
they say).</p>


Setting up a static website/blog with jekyll
2019-06-24T00:00:00+00:00
A couple of people asked me for more info on my new blog setup here's a rough
outline of what's involved in setup and posting. There are many</em> ways of doing
this, this is just the way that suited me best for now.</p>
What you need to follow along:</p>

A domain from GoDaddy</a></li>
An account on GitHub</a></li>
</ul>
Things I like about this setup</h1>

Built-in backups,</li>
offline editing,</li>
no shonkey web gui,</li>
no charge for hosting with custom domain and https,</li>
simple dev friendly templating,</li>
a nice simple code-friendly mobile-friendly design,</li>
full control.</li>
public revision
history</a> of blog
posts with no extra effort (no more "updated: blah" in blog posts)</li>
</ul>
Why jeykll fits that</h1>
A blog post like this one is just a text file in a special format called
markdown. These can then be trivially source-controlled with git, which serves
as version control and distributed backup in one neat package. I use git for
work so although it's not the easiest tool to use that's not a problem for me.
Theoretically if you don't know git you could probably just do all your editing
in the github web interface these days.</p>
Here's the entire</em> folder structure of this blog as it stands right now. You
can see there's really not a lot too it.</p>
tim@fox:~/repo/timwise.co.uk(master)</span></span>
$ tree</span></span>
.</span></span>
├── 404.md</span></span>
├── about.md</span></span>
├── categories.md</span></span>
├── CNAME</span></span>
├── _config.yml</span></span>
├── _drafts</span></span>
│   └── 2019-06-24-setting-up-a-jekyll-blog.md</span></span>
├── favicon.ico</span></span>
├── fonts</span></span>
│   ├── aramisi.ttf</span></span>
│   ├── EBGaramond-Regular.ttf</span></span>
│   └── FuturaPTLight.otf</span></span>
├── images</span></span>
│   ├── 404.jpg</span></span>
│   ├── reverie-demo.png</span></span>
│   ├── reverie.png</span></span>
│   └── reverie-text.png</span></span>
├── _includes</span></span>
│   ├── analytics.html</span></span>
│   ├── disqus.html</span></span>
│   ├── meta.html</span></span>
│   └── svg-icons.html</span></span>
├── index.html</span></span>
├── _layouts</span></span>
│   ├── default.html</span></span>
│   ├── page.html</span></span>
│   └── post.html</span></span>
├── LICENSE</span></span>
├── _posts</span></span>
│   └── 2019-06-21-yet-another-new-blog.md</span></span>
├── _sass</span></span>
│   ├── _darcula.scss</span></span>
│   ├── _highlights.scss</span></span>
│   ├── _reset.scss</span></span>
│   ├── _svg-icons.scss</span></span>
│   └── _variables.scss</span></span>
└── style.scss</span></span>
</span>
7 directories, 30 files</span></span></code></pre>
The posts are just text files using the markdown format with a .md</code> file extension instead of .txt</code> and a short header for information like page title and tags. Here's the top of this one:</p>
---</span></span>
layout: post</span></span>
title: Setting up a static website/blog with jekyll</span></span>
categories: [howto,backstage]</span></span>
---</span></span>
</span>
A couple of people asked me for more info on my new blog setup here's a rough</span></span>
outline of what's involved in setup and posting. There are *many* ways of doing</span></span>
this, this is just the way that suited me best for now.</span></span>
</span></code></pre>How I set it up</h1>
I actually am reconfiguring from another setup, and used some advanced
sillyness, but I'll simplify that here down to roughly the steps you'd need to
go from cold as I'm sure that's more interesting to you.</p>

Buy a domain from GoDaddy</li>
Create a GitHub account</li>
Find a jekyll
theme</a>
on become a base for your new site (the themes vary on how ready they are
for you to just crack on). Here's the reverie
theme</a> that I used with minimal
customisation.</li>
Go into the settings for your repo, </li>
turn on github pages, </li>
configure your domain name, </li>
enable https </li>
Go to GoDaddy and set up the DNS to point to the github
servers</a></li>
</ol>
Blogging with Jekyll</h1>

Create new text file in the _posts</code> folder, name it
yyyy-mm-dd-your-post-title.md</code>,</li>
steal the header block from above and customise it for your post,</li>
write stuff</li>
use markdown for layout,</li>
add images by saving them to an images folder and also adding them to your
git repo, use ![alt text](/images/something.png)</code> to show the image in your .md</code> file,</li>
check everything in & git push</code> to github,</li>
your changes are live within seconds of pushing.</li>
</ol>
If you aren't up to the full git</code> setup, you can edit your blog directly in github's web interface.</p>

Creating a new post
</li>
Editing a post
</li>
Editing the text of a post
</li>
</ol>
Advanced ninja coder blogging</h1>
Part of the reason I chose this particular setup is that plain markdown blogging with github pages allows me to use my existing coder tools to blog things really fast, reducing the barrier to me actually sharing useful insights and thoughts with you all.</p>
You can see the advanced editing flow in action here:</p>
https://www.youtube.com/watch?v=w3gMZTKcGKc</a></p>
</iframe>
Here's the tools I use, as shown in the above video:</p>

zsh</a> - allows changing directory without cd</code></li>
symlinks</a> to link blog folder into ~/blog</code></li>
shell script new</code></a></li>
template md file</a></li>
jekyll</a></li>
github pages</a></li>
github actions</a> - automatically deploys pages (automatically set up when you enable github pages)</li>
markdown</a> - this is important, it allows efficiently writing richly formatted blog posts with nothing but an (advanced) text editor such as vim or vscode</li>
vim</a> the original text editor (see also neovim</a> that I haven't got around to using yet)</li>
fzf the fuzzy finder</a> - so I can type **[tab]</code> and find any blog post easily</li>
vscode</a> with the vscode vim emulator extension</a> - ctrl-p to open any post quickly</li>
git</a> - provides history, and a very quick way to ship changes to the host (github) all on the command line</li>
I have many command line aliases in my dotmatrix</a> and aliases in my gitconfig</a> - these make working on the commandline (cli) muuuch faster.</li>
Ruby locally (advanced coder things)</li>
asdf-vm</a> to ensure I always have the right ruby version available on any machine, or can easily install it</li>
tmux</a> to allow me to have the server running in the background while I use the editor</li>
</ul>
⚠️ Note: ignore the _site</code> folder when choosing the file to edit, that's the generated cache and will be overwritten.</p>
Summary</h1>
Wordpress was overcomplicated for my needs, and slower to use/read, this new setup should mean I can tap away on the train and send new thoughts your way more regularly.</p>
Extras</h1>

Bonus points for MailChimp's rss to email</a> which means it'll automatically hit your inbox. (I haven't manageed to make it work yet though.)</li>
Here's all the jekyll gems github pages allows you to add https://github.com/github/pages-gem/blob/master/lib/github-pages/plugins.rb#L5</a></li>
</ul>


Blog refresh - bye bye wordpress, hello jekyll
2019-06-21T00:00:00+00:00
The old blog is gone. Move over wordpress, long live github
pages</a>.</p>
I haven't redirected the old subdomain or urls, but the stats tell me I had no
visitors so meh. If you want something let me know and I'll repost it.</p>
The new theme is : Reverie</a>.</p>
The full history and source of this blog is available in the open at
https://github.com/timabell/timwise.co.uk</a>.</p>


Database tools I was surprised existed
2019-06-10T00:00:00+00:00
But you already know all the tools for working with databases don't you?</p>
That's what I thought too before I started working on SQL Schema
Explorer</a>. Even after 18 years working with
databases it turns out I only knew a fraction of the tools that are out there.</p>
Working on SQL Schema Explorer gave me a reason I didn't have before to search
the internet in new ways relating to databases, and now that I'm reaching out
to people about SQL Schema Explorer people are in turn sharing their favourite
tools with me.</p>
I'd hate for this gift to sit hidden away in my private product trello board,
so I've turned it into this article so that you can skip the wilderness years
and gain a zen-like awareness of all the tools of your trade. I'm sure you
won't like or use all of them, and many overlap in function, but the following
list will allow you to try out and asses each one against your needs; perhaps
you'll find a new tool that will save you hours, or make your job that much
more enjoyable. Never again will you have to say to a new colleague "oh I
hadn't heard of that one!".</p>
Microsoft Sql Server</h2>
SSMS Tools Pack</h3>
https://www.ssmstoolspack.com/</a></p>
An add-on to management studio that improves many pieces of of ssms and adds new capabilities.</p>
{:height="200px"}</p>

€30 for first computer to €5000 for unlimited enterprise use (SSMS Toolpack
Pricing page</a>)</li>
</ul>
A friend of mine saved many hours using the "CRUD" stored procedure generation capabilities.</p>
Sql Server Management Studio (SSMS)</h3>
Okay you know this one but I have to mention it.</p>
It has awkward but functional diagram support. You can version control these
diagrams and move them between servers with
https://github.com/timabell/database-diagram-scm</a> which is worth knowing about
if you ever use the ssms diagrams.</p>
You can now run MSSQL on open source</h3>
Did you know Microsoft SQL Server (aka mssql) is now available on both linux
natively and in docker containers? It's the real deal, not like mono vs .net</p>

SQL Server on Linux</a></li>
SQL Server in Docker</a></li>
</ul>
Here's all it takes to fire up mssql, the only pre-requisite is docker itself.</p>
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=your_new_sa_passwod' \</span></span>
-p 1433:1433 --name mssql1 \</span></span>
-d mcr.microsoft.com/mssql/server:2017-latest</span></span></code></pre>
This isn't a tool per-se, but this is too important to not mention, especially
as it's quite new and not everyone is aware of this exciting change from
redmond.</p>
I don't know about you but one less reason to fire up the Windows VM sure does
make me happy. Combined with dotnet core I haven't fired up Windows in months
now.</p>
Cross-database</h2>
Razor Sql</h3>
{:height="200px"}</p>
http://www.razorsql.com/</a></p>

"RazorSQL is an SQL query tool, database browser, SQL editor, and database
administration tool for Windows, macOS, Mac OS X, Linux, and Solaris.
RazorSQL has been tested on over 40 databases, can connect to databases via
either JDBC or ODBC"</p>
</blockquote>
EZ Data Browser</h3>
{:height="200px"}</p>
http://www.softimum-solutions.com/Data-Browser/Overview.aspx</a></p>

$29.99 per computer</li>
Licence key by email</li>
pay via PayPal of credit card</li>
</ul>

"software from Softimum Solutions to help users to browse and to edit SQL Server databases quickly and easily"</p>
</blockquote>
Provides a configurable MSAccess-like interface to a database.</p>
SQLeo</h3>
Cross-database query builder/analyser. Feature rich - data/schema compare, pivot queries, diagrams etc.</p>
{:height="200px"}</p>

http://sqleo.sourceforge.net/</a></li>
</ul>

"A powerful SQL tool to transform or reverse complex queries (generated by
OBIEE, Microstrategy, Cognos, Hyperion, Pentaho ...) into diagrams to ease
visualization and analysis. A graphical query builder that permits to create
complex SQL queries easily. The GUI with multi-connections supports virtually
all JDBC drivers, including ODBC bridge, Oracle, MySQL, PostgreSQL, Firebird,
HSQLDB, H2, CsvJdbc, SQLite. And top of that, everything is open-source!"</p>
</blockquote>

SQLeo demo video on youtube</a></li>
Sourceforge page (ugh) https://sourceforge.net/projects/sqleo/</a></li>
Fork on github https://github.com/ojwanganto/SQLeo</a></li>
</ul>
SQL Fiddle</h3>
Live online sql editor / runner</p>
{:height="200px"}</p>

http://sqlfiddle.com/</a></li>
</ul>
DbSchema</h3>
{:height="200px"}</p>

http://www.dbschema.com/</a></li>
</ul>

"DbSchema is an SQL and No-SQL database designer featuring interactive
diagrams, HTML and PDF documentation, schema versioning and migration,
relational data browse, random data generator, visual query builder, SQL
editor and database reports."</p>
</blockquote>
DbPrompt</h3>
{:height="200px"}</p>

http://www.dbschema.com/dbprompt.html</a></li>
</ul>

"Free Universal Multi-Database SQL Prompt - DbPrompt can execute queries on
multiple databases, transfer data between databases, upload result files on
ftp servers, execute complex SQL scripts using Java Groovy and cron-schedule
scripts for execution and report failures per email. DbPrompt supports all
SQL and NoSQL databases, like MySql, Cassandra, PostgreSql, MongoDb,
Redshift, SqlServer, Azure, Oracle, Teradata and more. DbPrompt can work on
all operating systems. DbPrompt is free of charge. "</p>
</blockquote>
Unsorted - omg will it never end</h2>

http://www.magereverse.com/</a></li>
https://dataedo.com/</a></li>
https://dbvis.com/</a></li>
https://docs.microsoft.com/en-us/sql/azure-data-studio</a></li>
https://github.com/ajdeziel/SQL-Data-Viewer</a> - abandoned</li>
https://github.com/preston/railroady/</a></li>
https://help.talend.com/reader/ISPDm8GQ6s0HN0348QulWg/Ij~7tBlW8im63rAGnGHT3A</a></li>
https://portableapps.com/apps/development/database_browser_portable</a></li>
https://redash.io/</a></li>
https://sequelpro.com/</a></li>
https://softwarerecs.stackexchange.com/questions/11346/tool-to-visualize-sql-database-schema</a></li>
https://sqldbm.com/en/</a></li>
https://sqlitestudio.pl/</a></li>
https://www.codediesel.com/data/5-tools-to-visualize-database-schemas/</a></li>
https://www.datasparc.com/</a></li>
https://www.dbsoftlab.com/online-tutorials/active-table-editor-online-tutorials.html</a></li>
https://www.devart.com/dbforge/sql/studio/</a></li>
https://www.idera.com/er-studio-data-architect-saftware</a></li>
https://www.jetbrains.com/datagrip/</a> - https://www.youtube.com/watch?v=Xb9K8IAdZNg</a></li>
https://www.metabase.com/</a></li>
https://www.navicat.com/en/products/navicat-data-modeler</a></li>
https://www.schemacrawler.com/</a></li>
https://www.sqlservercentral.com/articles/microsoft-sql-server-utilities-and-tools-1</a></li>
https://jetbrains.com/</a> intellij/rider/rubymine etc all have the datagrip capabilities built in</li>
https://www.red-gate.com/</a> sql prompt etc</li>
https://www.oracle.com/database/technologies/appdev/datamodeler.html</a></li>
https://erwin.com/products/erwin-data-modeler/</a></li>
DBeaver</li>
Azure Data Studio - PostgreSQL extension</li>
SAP PowerDesigner https://www.sap.com/products/powerdesigner-data-modeling-tools.html</a> - I have no idea what this is. Sounds enterprisey.

https://en.wikipedia.org/wiki/PowerDesigner</a></li>
</ul>
</li>
https://www.idera.com/er-studio-enterprise-data-modeling-and-architecture-tools</a></li>
https://www.idera.com/er-studio-data-architect-software</a></li>
https://sparxsystems.com/products/ea/</a></li>
https://www.ibm.com/us-en/marketplace/infosphere-data-architect</a></li>
https://www.postgrescompare.com/</a></li>
</ul>
Places to find even more database tools and learn more</h2>

https://en.wikipedia.org/wiki/Comparison_of_database_tools</a></li>
https://www.quora.com/How-do-I-generate-an-entity-relationship-diagram-for-a-SQLite-database-file?share=1</a></li>
https://www.quora.com/What-are-some-good-online-database-schema-design-tool-with-larger-days-of-expiry</a></li>
https://alternativeto.net/software/mysql-workbench/</a></li>
https://www.datasciencecentral.com/profiles/blogs/top-6-data-modeling-tools</a></li>
https://www.educba.com/9-best-data-modeling-tools/</a></li>
http://www.agiledata.org/essays/dataModeling101.html</a> - I had no idea what data modelling really meant before reading this.</li>
http://www.sqlservercentral.com/articles/Tools/157911/</a></li>
https://www.apgdiff.com/</a></li>
http://toolsfordatabases.com/</a></li>
https://www.webfx.com/blog/web-design/top-five-best-database-management-tools/</a></li>
https://solutionsreview.com/data-management/data-management-solutions-directory/</a></li>
https://www.softwaretestinghelp.com/tools/26-best-data-integration-tools/</a></li>
</ul>
Information overload</h2>
If you got this far you probably need to get back to work. It's clear this
article is a bit ridiculous now so I'm not going to carry on sorting it out.
The internet is littered with catalog articles of varying completeness and
quality. To make it complete and good would make it dull and endless.</p>
Watch this space, I think I'll try and work out what people actually are trying
to do and cater for that. Maybe we can all build a collaborative index
somewhere.</p>
The end</h2>
I hope you found at least a few you didn't know about and that they make your
life better in some way. Please do tell me the story of how this helped you on
email</a> or twitter</a>.
Did I miss something? If you wish to improve this article please ping me a PR
with additions here: https://github.com/timabell/sdv-website or just email
me</a>.</p>
I'm not being paid to promote these, these are not affiliate links, I share
this learning with you all for free so that we can all enjoy our work with
databases more, and create better more reliable databases for ourselves, our
clients and our projects.</p>
If you want to be notified of new articles, sign up to the mailing list (which
currently is also the trial download list).</p>
Till next time!</p>
Originally posted at http://schemaexplorer.io/blog/2019/06/10/database-tools-you-didnt-know-about.html</a></p>


Teardown of a landing page story
2018-12-10T08:40:13+00:00
Here's my analysis of the depth of information and thought that's gone into a deceptively brief piece of landing-page copy. It's from https://realtimeboard.com/blog/accelerate-growth-early-stage-startups/</a> that I'm trying to use as inspiration for http://schemaexplorer.io/</a></p>
I had to de-construct the example story in order to work out how to do my own so having put the work in I thought I'd share it with you all here.</p>

Here's the original in full:</p>

“Hey, I know that you have two kids and you’re always buying the same milk, eggs, peanut butter, and bread every single week, but you’re also both really busy working parents. You don’t have much time and you’d much rather be able to spend time at home with your family than being at a busy grocery store. So there’s this great service called Farmstead where you’re able to have that same stuff delivered to you. You don’t even have to think about it, you don’t have to remember each week to put in all of those same things that you order on a weekly basis anyway. They can just bring it to you and you don’t have to worry about it. Delivery is free with a weekly subscription. I think you should definitely try this out, I think you’re going to save yourself a bunch of time.”</p>
</blockquote>
And now one piece at a time with my observations about what is being conveyed in this information-dense block of prose. You are looking at the work of a pro when you read the above text, and unless you look closely it's easy to miss just how carefully crafted the whole thing is and how it touches on so many thoughts and feelings the ideal customer of Farmstead is likely to have, guiding them to the conclusion that they should give the service a try.</p>


Hey,</p>
</blockquote>
being personable, we are not a big faceless corporation, you can actually talk to us like humans</p>

I know that you have two kids</p>
</blockquote>
narrow down audience to parents (turn off non-parents, get parents more interested)</p>

and you’re always buying the same milk, eggs, peanut butter, and bread</p>
</blockquote>
describe a behaviour they recognise they already do (increase engagement with the text, encourage reading further), hint that we're going to solve a problem they have because they might be annoyed about the regular chore</p>

every single week,</p>
</blockquote>
slightly pejorative terminology hints at the fact you might not be happy about repeating this tedious task, draw attention to your current pain that we're going to solve</p>

but you’re also both really busy working parents.</p>
</blockquote>
focus down the pain point to be that it's a time consuming repeating the task and you wish you had more time back, hint that we're going to be giving that time back to you</p>

You don’t have much time</p>
</blockquote>
really drive that you have no time</em> point home</p>

and you’d much rather be able to spend time at home with your family</p>
</blockquote>
paint a picture of how much better your life would be if you take on our solution to your pain. Point out just what you're sacrificing to go to the shops.</p>

than being at a busy grocery store.</p>
</blockquote>
not only do you have the pain of lost time, but the experience of shopping is unpleasant (because it's so busy)</p>

Now that you know it's you we want to help, we recognise your pain and we've shown we know how your life could be better...</p>

So there’s this great service called Farmstead</p>
</blockquote>
Introduce the brand name. Tease. Read on to satisfy that completion-bias tendency in your brain that leaves you yearning to know what information you don't have about this "Farmstead".</p>
Also the name itself conveys a lot of meaning</p>
"Farm" - fresh produce, direct from the supplier
"stead" - homely, small supplies, not industrial or full of chemicals</p>

where you’re able to have that same stuff</p>
</blockquote>
allay the fear that you'll still have to go back to the old thing anyway because there's something we don't sell</p>

delivered to you.</p>
</blockquote>
oh I see, so it's a door to door delivery service. Noted. The shape of the offering is starting to emerge.</p>

You don’t even have to think about it,</p>
</blockquote>
We've worked really hard on all our UX (user experience), automated as much as possible etc. so this will be easier than what you do now and better than competitors can do</p>

you don’t have to remember each week to put in all of those same things that you order on a weekly basis anyway.</p>
</blockquote>
now you know it's even got built in scheduled deliveries</p>

They can just bring it to you</p>
</blockquote>
re-iterate delivery</p>

and you don’t have to worry about it.</p>
</blockquote>
reminder that you do have have to worry about getting your normal shop done, speaks to pain of running out of the things because you forgot, worrying how you're going to fit it in your day. Differentiates new offering as being something that just happens meaning you never run out</p>

Delivery is free</p>
</blockquote>
Objection handling - people don't want to pay for delivery when they could go to shops for nothing (even though it has a cost it's how people see it). People have been burned before by finding something good then being stung for expensive delivery. I would expect that this is a common objection/question that came back in earlier rounds of testing with users and it's been integrated into this prose to pre-empt the question before it's even been asked.</p>

with a weekly subscription.</p>
</blockquote>
warm people up to the payments they'll have to make in a non-threatening way</p>

I think you should definitely try this out,</p>
</blockquote>
nudge to take action. "Try" implies that it's not a commitment and you can change your mind, allaying fears of being locked in for a year</p>

I think you’re going to save yourself a bunch of time.</p>
</blockquote>
point to hard cost-benefit for the service - you don't have time (as mentioned to start with, we know this person feels like they have no time because they are still reading after they identified with the first section). One last reminder of the key pain to the offering solves - lack of time</p>

Let me know if that's helped you in any way.</p>
If you happen to be someone that deals with databases and are keen to try new tools then please do me a favour and sign up to the http://schemaexplorer.io/</a> mailing list (you'll have to ignore my terrible marketing text till I'm as good as Brady).</p>
~ Tim</p>


A Modern Digital Government Outsourcer
2018-10-30T14:19:26+00:00
I've done a few contracts for our dear government and an idea for a business is starting to form. Here is the first draft of these ideas as a starting point for discussion and improvement.</p>
The problem as I see it</h2>
From my somewhat limited perspective as someone who has worked on a few software projects for government, including having the privilege of a brief stint with GDS</a> (wikipedia page on GDS</a>), I can see a looming problem of capacity that I wonder if I can help with.</p>
A change is happening to the way government runs digital projects (or entire services with a digital aspect I should say). This change was kicked off with the formation of GDS back in 2011, and has been rippling out</strong> across the many departments of government ever since.</p>
The skills</strong> to run such projects are in ever increasing demand as more politicians and senior civil servants realise the effectiveness of the approach and are pushed into the approach by the controls made available to GDS through the Cabinet Office.</p>
Some of the projects going on in government a resourced with existing internal people</strong> (many of whom are contractors) who have been moving around departments executing brilliantly in the true spirit of who GDS started, but there are only so many of these people to go around.</p>
There are consultancies who "do agile"</strong>, but that is only part of the picture of spinning up a new digital-focussed project and having it succeed.</p>
And then there's the big old outsourcers</strong> who have for decades been the go-to for large government IT. It strikes me that the old-world outsources struggle to meet the needs of the new world, as it's a fundamental shift away from outsource-all-the-knowledge / lowest-bidder-wins / change-requests-for-profit model that to be fair they have been incentivised to operate within.</p>
All of this leaves ministers and civil servants who have seen how well it can be done wondering where on earth they can find the people they need to make a project a success. It's all very well having a service standard</a> that advocates a modern user-centred service but if you can't rustle up a team or outsourcer who can make that happen then you are stuck in the old world with broken, overpriced and inflexible IT systems that we are so used to.</p>
Note that the GDS way is not a refusal to use outsourcers at all, rather it is a call to not be held to ransom by outsourcers by passing all skills and knowledge over to them wholesale and them being unable to control them.</p>
A new breed of outsourcer</h2>
Here I'm going to set out a rough design for a business that could help senior people in government deliver these services. Your feedback and ideas would be greatly appreciated as I cannot create this alone.</p>
The business would take leadership</strong> in researching, designing, iterating and building services where a department currently lacks the capacity to do so but has the desire to build a modern service for some perceived user need. It would do this by supplying one or more people who truly understand the GDS-way, and will guide the service through the cycles of discovery and delivery.</p>
The business would explicitly not</strong> try to capture ownership, insider knowledge or advantage over the department in order to gain control over the service. Instead the business would ensure that the department receives the benefit of learning that happens during the project, producing outputs that can be referred to in the long run, and ensuring that relevant people within the department are (with the assistance of the department) deeply involved in the process.</p>
The business would attain speed of delivery by recruiting and training a team to be able to assist the department in conducting research and delivering services. This team would consist of mixture of senior people who can make sure the GDS way is followed as much in spirit as in letter, and a deeper pool of more mixed experience who can be guided by the senior team members in delivering effectively in a way that works for the department, GDS, and the users. The business shall build processes around this team that assist them in repeatably recruiting, training and scaling while retaining the core principles that work in the favour of government and our users.</p>
The department will be encouraged (maybe even required?) to build internal capability (i.e. have enough smart skilled people) to retain the knowledge gained during the process of building a service (or perhaps learning that it doesn't make sense to build a particular service). The department will also be encouraged to actively recruit team members to bolster their ability to run the services in the long term and retain the knowledge gained as a service is created and evolved (perhaps for an appropriate fee to compensate for the investment in the people who pass through the business's doors).</p>
The business needs to be a profitable enterprise in order to be a sustainable entity that can help our government in the long run. Suggestions for funding models most welcome!</p>
The business will not retain any assets / intellectual property etc. would allow the government to be held to ransom by the business. The government will always have the option to take the whole service in-house, or choose another outsourcer. Support for transition will gladly be provided (for a reasonable fee). The government is encouraged to own its own services.</p>
How are you different from public digital</a>?</h2>
Public Digital https://public.digital/ are a business founded by the founding members of GDS, and are off transforming other governments and organisations from scratch. That's a lofty goal and I wish them well! My goals are more modest, simply to provide the capacity, processes and skills pool for government deliver on the vision and habit of delivery left in the wake of these powerhouses of change. I hope I can make even a thousandth the impact they have made.</p>
How to get there from here</h2>
Currently I'm contracting, I like the idea of slowly growing a team from here to there. I'm not in a position to just drop everything and gamble everything on success, but perhaps there can be a smoother transition. For now I just want to have as many conversations as possible, so get in touch</a> and let me know how you feel about the whole idea. Love it? Hate it? I want to know.</p>
Get involved</h2>
Drop me an email: tim@timwise.co.uk</a></p>
Sign up to the mailing list: http://eepurl.com/dLU4Rs</a></p>


Always add braces
2018-08-28T12:14:02+01:00
Small matter of code style that I keep coming across.</p>
Whether to write</p>
if</span> (</span>thing</span>)</span> Action</span>(</span>)</span>;</span></span></code></pre>
or</p>
if</span> (</span>thing</span>)</span></span>
  Action</span>(</span>)</span>;</span></span></code></pre>
or</p>
if</span> (</span>thing</span>)</span></span>
{</span></span>
	Action</span>(</span>)</span>;</span></span>
}</span></span></code></pre>
We should always use the braces to avoid introducing bugs when modifying the code manually or through merge tools.</p>
For the reasons behind this, read the following:</p>

https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/</a></li>
https://dzone.com/articles/omitting-braces-not-just-a-mat</a></li>
Anatomy of a “goto fail” – Apple’s SSL bug explained, plus an unofficial patch for OS X!</a></li>
</ul>


5 ways to make your database better - by Tim Abell
2018-07-10T00:00:00+00:00
[1] Documentation</h2>
Shoot me okay, but maintenance of software is [insert large number here] times
the cost of creation, especially with relational databases. You are a pro
working for a client, you owe it to them to make it possible for them to have
future staff (and yourself!) be as effective as possible. You put all that
effort into figuring out why a column should exist and have that name, now
share that knowledge before you move on to the next greenfield project</p>

Redgate SqlDoc</a>
is great for rapidly adding missing documentation.</li>
SchemaSpy</a> generates static html sites making it easy
to see what documentation there is (or isn't!) and share it with the team.
It's free & open source (although a bit fiddly to set up and run). It has
particularly nice clickable diagrams.</li>
Dataedo generates static html sites &
pdfs</a>
as well, and is commercial and slicker than SchemaSpy</li>
This gist for source-controlling ms_description
attributes</a> gives you
a two-way source-controllable / editable list of your documentation in SQL
Server</li>
SQL Schema Explorer</a> generates dynamic html sites
making it easy to see what documentation there is and share it with the team.</li>
</ul>
[2] Refactor your database</h2>
Migrations are a thing now. Use them. You refactor your code, why wouldn't you
refactor your database? Stop leaving landmines for future people - misleading
names, bad structures etc. Use the redgate tools (ready-roll etc), use your
orm’s tools (EF migrations, active record migrations). Yes you have to deal
with data, but it’s the exception not the rule that it’s going to take hours to
run because of data volumes.</p>

Redgate's SQL Change
Automation</a>
(formerly
ReadyRoll</a>)
is an opinionated tool for creating and running database migrations, it even
generates Database Administrator (DBA) friendly pure-sql deployment packages.
Very impressive!</li>
Redgate's SQL Source Control supports
migrations</a></li>
I've been using EF Core
migrations</a>
recently and they work well. There are equivalents for all the major
platforms.</li>
</ul>
[3] Enforce data integrity</h2>
Does your app fall over if the data is bad? Databases have many powerful ways
of enforcing the rules your code relies on: nullability, foreign keys, check
constraints</a>, unique constraints.
Stop the bad data before it even gets in there. Now your database is enforcing
these rules your code doesn't have to handle violations of them when reading
data because they'll never happen</p>
[4] Integration testing</h2>
You have an ORM. Great. You have unit tests. Great. But where the rubber hits
the road and your code sends SQL to a real database it breaks at runtime more
often than you’d like to admit because the generated sql didn't jive with the
real database structure or data in some obscure fashion. Automate the
creation/test/destruction of your db and run full end to end integration tests.
I suggest automating from the layer below the UI to keep the tests fast. There
are many techniques for keeping the tests quick but still realistic: do end to
end smoke tests instead of individual pieces, use an in-memory database, use
database
snapshots</a>
or the fancy sql-clone</a>
tool from Redgate to make creation / rollback virtually instant. Can you pull
realistic (anonymised) data from production? Better still, now you’ll catch a
whole new class of bugs before they hit prod.</p>

Here's a guide from Redgate detailing one way to do continuous integration
testing with
databases</a></li>
</ul>
[5] Make it visible</h2>
Are the only people that can see the database structures the coders and DBAs?
do the business owners, support people, Quality Assurance (QA) people find it a
mystery? You should be just as proud of your database as you are of your code,
by shining a light on this dark corner of your digital estate you can make it
as good as it should be, not an embarrassing backwater. By sharing the database
in an accessible form to the non-coders in your team you can help them be more
effective in their jobs.</p>

The html generated by SchemaSpy</a> can be shared on any
webserver to let your whole team see your schema structures</li>
SQL Schema Explorer</a> can be run on your network
or cloud hosting (schema explorer is
dockerized</a>!) to give your team easy
access to both the schema and data within the database.</li>
</ul>
Combine these tools with a continuous integration system and you have easy
access to the bleeding edge of your databases development.</p>
Take action now!</h2>
1.Make a start on at least one of these improvements today.
2.Share this article with your team - get everyone motivated to improve.
3.Share this article on social media - help spread the word that our
databases deserve better!</p>
I hope this has inspired you to make an improvement in the often unloved
underbelly of your applications.</p>
What do you think needs improving in the way we deal with databases? What
change did you make because of this? [Let me
know!](mailto:articles@timwise.co.uk?subject=making better databases)</p>
Originally posted at
http://schemaexplorer.io/blog/2018/07/10/5-ways-to-make-your-database-better.html</a></p>


Building on my own just for you and your database
2018-06-26T00:22:08+01:00
I'm not a big company, in fact it's just me. I haven't even paid anyone to do anything other than my accountants. I'm not against the idea of paying people, I just want to know I really need to first.</p>
I see a gap in the tools available to database people; it's bothered me for years (decades actually, I built an in-house variation of this in Classic ASP in the year 2000, yeah I'm getting old, thanks for pointing that out). I'm not the only one who recognises this gap, someone I showed schema explorer to said "oh I knew someone who built something like that", this is great as it means I'm not delusional about there being a problem to solve. I find it hard to explain to people what this thing is, as there's nothing quite like it out there. I'm still not really sure who it's for (other than previous visions of myself). I want to put the time and effort in to make something that's so valuable that people are happy to pay for it, and the money that brings will in turn allow my work to continue on the tool, bringing benefit to more people.</p>
The core of the idea was navigating data across foreign keys, after talking to people it's evolved into an all-round schema (and data) exploration tool (hence the addition of diagrams). My vision is to keep it a general-purpose database-centric tool, avoiding temptation to solve niche problems with it where it could take away from that well-formed-tool experience. I think I now have a piece of downloadable software that's basically useful as it stands; but that there's something (or many things) between where it's at now, and people starting to get value from it by using it with their own databases. I don't know yet what that is and I hope to find out through more conversations with database people. I certainly have plenty of ideas what it could be, but need to collect evidence rather than trying to fix non-problems on the off-chance. Especially with my even more limited time now that we've just had our second baby boy.</p>
This is my current story, I hope you found it more interesting than the world cup ⚽.</p>
http://schemaexplorer.io/</a></p>


Building a Windows 10 Development VM from scratch
2018-04-01T20:10:54+01:00
Rebuilding windows dev VM from scratch, reminds me why people put up an old build for so long. What a painful process!</p>
I did wonder if I could get away with mssql on linux, and tried the mssql docker image, which hosed my linux kernel</a>. That's too many yaks for one morning.</p>
Windows Install</h2>
Download a virtualbox image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/</a></p>

Attach the vm disk in virtualbox</li>
Fire up and run updates</li>
Install Rider via jetbrains toolbox</li>
Shut down the VM</li>
Take a virtualbox snapshot</li>
</ul>
backing up a vm</h1>
cd VirtualBox\ VMs/  </span></span>
base=.  </span></span>
src=win10-2018  </span></span>
ls -lh $src  </span></span>
du -sh $src  </span></span>
df -h .  </span></span>
tar -cpC $base $src -P | pv -s $(du -sb $base/$src | awk '{print $1}') | lz4 >> $src.tar.lz4  </span></span>
# 8.79GiB 0:00:43 [ 205MiB/s] [=====================================================================> ] 99%  </span></span></code></pre>
Gist for backing up with lz4</a></p>
Updates! Updates! Updates!</h2>
(as Ballmer once said?)</p>

poke windows updates</strong> until it finally decides there's no more</li>
reboot</li>
poke windows updates until it finally decides there's no more</li>
reboot</li>
poke windows updates until it finally decides there's no more</li>
reboot</li>
Install virtualbox guest additions 5.1.34</strong></li>
reboot</li>
shutdown</li>
backup again</li>
</ul>

</a>
</div>
The Snapshot Trials</h2>
VirtualBox has a neat snapshot feature</a>, so you can try out a whole tree of attempts at installs then squash them into the final image or roll them back.</p>
Take a snapshot as a starting point for trying out the eternally broken boxstarter script.</p>
Boxstarter</h2>
Boxstarter script, mostly working in win 10</a> now, but so much more to do on top that, including windows branch of dotfiles</a> (always needs work). Read the script to see what it configures and installs.</p>
You'll want to watch this vid at at least 2x playback speed:</p>
https://youtu.be/Hiz9_i67B3o</a></p>
annoyances / setup - grab reg keys</h2>

disable screensaver</li>
hide task manager when minimized

HKCU\Software\Microsoft\Windows\CurrentVersion\TaskManager\Preferences</code> - binary. sigh.settings > m</li>
</ul>
</li>
multitasking > snap (these have been rolled in to my boxstarter, but not tested there).

HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced\SnapAssist</code> - 0

HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced\JointResize</code> - 0</li>
</ul>
</li>
all sounds off - generates 100s of reg keys, one for each event</li>
mute</li>
</ul>
get ready for dev</h2>

vs sign-in</li>
extension updates

ssdt - external download popup, installs sql server 2016 localdb, takes forever, requires reboot</li>
azure - leave for now, also external</li>
</ul>
</li>
jetbrains toolbox (sign in)

resharper</li>
datagrip</li>
</ul>
</li>
</ul>
VS Extensions</h2>
cinst editorconfig.vs vsvim ihateregions -y</code> - bit-rotten, didn't work - I should contact the VsVim package maintainer</a>. Oh never mind that would be me then.</p>
Extensions to install manually:</p>

https://marketplace.visualstudio.com/items?itemName=JaredParMSFT.VsVim</a></li>
https://marketplace.visualstudio.com/items?itemName=ZoltanKlinger.RelativeLineNumbers</a></li>
https://marketplace.visualstudio.com/items?itemName=MadsKristensen.MarkdownEditor</a></li>
https://marketplace.visualstudio.com/items?itemName=VisualStudioProductTeam.VisualStudio2015ColorThemeEditor</a> - to get solarized colours</li>
</ul>
Something to look in to https://chocolatey.org/packages/batch-install-vsix</a></p>
Git setup</h2>
GitExtensions defaults to the program files putty, but the chocolatey one is newer and github deprecated the comms the old one uses.</p>

https://github.com/desktop/desktop/issues/4105</a></li>
https://blog.github.com/2018-02-23-weak-cryptographic-standards-removed/</a></li>
</ul>
Change all the putty paths in git extensions to start with C:\ProgramData\chocolatey\bin\...</code></p>
![Workspace 1_193]({{ site.baseurl }}/assets/workspace-1_193.png)</p>
https://git-extensions-documentation.readthedocs.io/en/latest/settings.html</a></p>

Settings that are specific to Git Extensions and apply globally will be stored in a file called GitExtensions.settings</code> either in the user’s application data path or with the program. The location is dependant on the IsPortable setting in the GitExtensions.exe.config</code> file that is with the program.</p>
</blockquote>
i.e .C:\Users\tim\AppData\Roaming\GitExtensions\GitExtensions</code></p>
Key Setup</h3>

Map a read-write folder to outside the VM to keep the putty keys in (this avoids regenerating the key if the VM is regenerated / rolled back, and also means keys aren't copied around with the VM image which makes them a bit easier to keep track of</li>
git extensions > tools > putty > generate</li>
set passphrase</li>
save pub & private keys to shared folder</li>
grab the public key from the generator, paste into github & bitbucket website account ssh configs</li>
git extensions > tools > putty > start agent > add keys</li>
</ul>
DotMatrix</h3>
Start git bash from git extensions, not the start menu (to get ssh set up right).</p>
cd /c/repo && mkdir tim && cd tim  </span></span>
git clone git@github.com:timabell/dotmatrix.git  </span></span>
cd dotmatrix/  </span></span>
git checkout windows  </span></span>
bin/install  </span></span>
git st  </span></span></code></pre>VS Settings</h2>
![Workspace 1_194]({{ site.baseurl }}/assets/workspace-1_194.png)</p>
VS > Options > Environment > Import/export > set the path to the dotmatrix copy</p>
resharper settings</h2>
![Workspace 1_195]({{ site.baseurl }}/assets/workspace-1_195.png)</p>
VS > Resharper > manage options > right-click the gap > add layer > open settings file > select the one from the dotmatrix</p>
More settings</h2>

VsVim settings</li>
git config</li>
email/name</li>
</ul>
More installs</h2>
redgate tools</p>

dev bundle:

sql prompt - for VS too</li>
compare</li>
data compare</li>
doc</li>
</ul>
</li>
toolbelt bundle

readyroll</li>
</ul>
</li>
</ul>
firefox</h2>

sign-in</li>
ghostery settings - disable annoying purple dot</li>
</ul>


My Screencasting Setup
2018-03-17T18:56:12+00:00
I just made a video to demo schema explorer.</p>
</iframe>
I did this in one take with no post-production work. I figure practice will improve my selection and delivery of the content.</p>
Behind the scenes I used the following:</p>
Hardware</h2>

Dell XPS 13 - (9350 model), using the built-in webcam (up-nose view, camera in bottom left of lid for some reason), good enough for tiny pic in corner.</li>
Snowball usb mic</a> (great sound quality, no worrying about analogue stuff), sat on a folded towel on my desk to deaden vibrations through the mini-tripod. The snowball has 3 settings on a switch at the back, had it on 3 - omni directional, I tested all three modes</a> beforehand.</li>
Supplied stand didn't actually fit properly, so bought an ammoon mini tripod</a> instead which is lovely (importantly it has the right screw thread for the mic).</li>
A quiet room after everyone is in bed to avoid interruptions</a>, and did the recording between rain storms!</li>
</ul>
Software</h2>

Linux Mint</a> 18.3 Cinnamon</li>
Vokoscreen</a> 2.4.0 - this handles screen recording & the picture-in-picture from the webcam</li>
</ul>
I recorded the entire screen at 1920x1080 (seeing as even phones are HD now). and uploaded it to youtube. (drag-n-drop, simplez!)</p>
Settings</h2>
Control panel > sound > input > snowball > about 130% input volume</p>
![Sound_183.png]({{ site.baseurl }}/assets/sound_183.png)</p>
fullscreen capture, display 2, 1920x1080</p>
![Screenshot from 2018-03-17 18-27-39.png]({{ site.baseurl }}/assets/screenshot-from-2018-03-17-18-27-39.png)</p>
pulse / blue-snowball selected</p>
![Screenshot from 2018-03-17 18-27-55.png]({{ site.baseurl }}/assets/screenshot-from-2018-03-17-18-27-55.png)</p>
mpeg4, mp3, avi, 25fps</p>
![vokoscreen 2.4.0_180]({{ site.baseurl }}/assets/vokoscreen-2-4-0_180.png)</p>
Default config apart from save path "videopath"</p>
![vokoscreen 2.4.0_181.png]({{ site.baseurl }}/assets/vokoscreen-2-4-0_181.png)</p>
Webcam switched on</p>
![vokoscreen 2.4.0_182.png]({{ site.baseurl }}/assets/vokoscreen-2-4-0_182.png)</p>
Other attempts</h2>
I spent ages finding the right software and doing test takes to get sound levels etc right.</p>
I started off with Kazaam for recording and mplayer for the webcam onscreen video.</p>
I got mplayer to show a borderless webcam video picture on screen that I could then use with any screen recorder, but had problems with dual screen behaviour so even though I could zoom in better I gave up on that once I'd discovered vokoscreen.</p>
You might also want to look at https://wistia.com/soapbox</a> which I hear is very good.</p>


Regression Tests for ASP.NET / SQL Projects
2018-03-13T15:25:46+00:00
I've had the opportunity to attempt many variations of regression testing on many projects.</p>
Many of the systems I've worked on have at least some level of workflow, meaning a user has to move through many steps as the system progresses something towards a final state.</p>
As systems become more and more complex it becomes harder to not accidentally break an area of the system you weren't currently looking at and didn't re-test manually before shipping. Hence the search for the holy grail of the perfect regression test suite. It turns out that none of the approaches available (unit tests, UI drivers etc.) provide a single solution to the problem, and instead we have to use all these approaches in balance to give reasonable protection against regressions.</p>
Here I lay out my current thinking on the best way to use testing to avoid regressions in a standard ASP.NET MVC + SQL Server project. I would like to introduce to you the idea of "business logic tests" which are an integration test suite starting below the UI layer, continuing through to a real database. I believe this gives an ideal balance of the trade-offs we have to make when choosing how to design out full regression test suite for a system.</p>
System layers</h2>
The layers of such a system are ideally something like this:</p>

Database (largely just used for CRUD storage)</li>
ORM - EF or similar library</li>
Model - C# classes</li>
Business logic layer - pure C# - (often missing in real projects, with business logic scattered across the other layers)</li>
UI layer - Controller classes written in C#, views in razor .cshtml files, html, css, javascript etc.</li>
</ul>
Recommended test layers</h2>

Unit tests</strong> - test small pieces of C# code</li>
Business logic tests</strong> - test the business logic layer down through to a real SQL Server database.</li>
UI Smoke tests</strong> - non-exhaustive test of a few common user journey's through, running against a real database, driven with Selenium Web Driver or similar.</li>
</ul>
There should be only a few smoke tests, there should be many business logic tests covering all business functions that need to stay working, and finally developer discretion on the amount of unit testing to apply at the lowest level.</p>
What do the business logic tests look like?</h2>
These could be pure code driven through something like NUnit, or they could be in business language through a BDD tool like specflow.</p>
They depend on the business logic project but nothing above that (UI). They inject a real database connection as a dependency through whatever injection method you use.</p>
These tests should make sense to the product owner / client.</p>
This is the layer with the broadest coverage of all of the three styles in play. Every business use-case, workflow & variation should be covered here. This is the suite that will alert you if any business or workflow rule is no longer being applied as designed, or if an obscure step of an obscure branch of your workflow has stopped working (perhaps due to an odd interaction with a database constraint).</p>
Handling the database</h2>
SQL Sever has a very useful snapshot feature that means you can very rapidly roll back to a clean state after each test run. (Snapshot setup script</a>)</p>
For each business logic test run:</p>

Reset to the production snapshot</a> (or production equivalent build if you can't actually grab a production backup)</li>
Run outstanding migrations for this version of the code (you are using Octopus + ReadyRoll</a> aren't you?)</li>
Run the Business logic tests.</li>
</ol>
Why not test all the business functions through the UI?</h2>

UI tests are prone to reliability problems, you can end up with 98% pass rates being normal, so then you don't notice when a critical test fails.</li>
UI tests are slow - this can add an extra overhead to developer productivity.</li>
UI tests cause unwarranted resistance to change - once you have UI tests the UI becomes extremely hard to change, particularly for sweeping site-wide redesigns without spending a prohibitively long time fixing up the test suite. And you aren't fixing the test suite because you changed the logic, this is busywork.</li>
The UI isn't sufficiently fragile to warrant this level of testing - when you are working on a page you are unlikely to break unrelated pages without realising.</li>
</ul>
I have seen first-hand a team of excellent programmers get bogged down in problems with a full regression test suite driven entirely from the UI.</p>
Why not test all layers separately?</h2>
In my experience a major point of fragility when a system is modified is the interaction between the code and the database. As such it's important that this fault-line is covered thoroughly by all regression tests that focus on business functions that need to remain functional.</p>
A test suite that isolates the business logic layer from the database layer provides very little value, usually just fulfilling a "though shalt unit test all classes</em>" declaration.</p>
How to get there</h2>
If you don't have a solid business logic layer then you will need to refactor your way to that goal. This has other benefits for reliability beyond the ability to test. Clean code with an obvious layer for all the business logic is easier to modify and keep defect-free.</p>
Bonus - free test data</h2>
The tests that you write to drive the business logic layer will be generating real data in a real copy of your database as it runs through the many steps in a workflow. You can use this test driver to generate useful test data for every step in a workflow, e.g. if a workflow had four steps you could do the following:</p>

Test account A - drive business logic to step 1</li>
Test account B - drive business logic to step 2</li>
Test account C - drive business logic to step 3</li>
Test account D - drive business logic to step 4</li>
</ol>
This is extremely quick to run, and now you have an account in every state of your workflow. This is useful for easily:</p>

Demoing your system from any starting point with multiple test accounts.</li>
Having the data in the right state to do development work on state 4 without having to manually set it up.</li>
Give your manual testers all the test accounts they need in many interesting states.</li>
Testing out your reporting systems and anything else that consumes data from your database.</li>
</ul>
Having realised you can do this, why would you ever do it manually again?</p>
Because the code that generates this data is a core part of your regression test suite it will be maintained as part of normal development activities and so you'll never again have to deal with test data that no longer works with your latest version of the system.</p>

If you want me to help you get to this ideal then drop me a line - tim@timwise.co.uk</a></p>


Developers, Love Your SQL Database
2018-03-12T19:30:15+00:00
Developers, don't be afraid of your SQL database, don't try and ignore it, try and make it the best it can be just like you do for your code.</p>
What's wrong today</h2>
I've noticed that a lot of projects that have SQL databases don't apply the high standards that they have for their codebase (refactoring, clean code, documentation, etc.) to the SQL part of their projects.</p>
Dealing with relational databases is harder than code:</p>

The tooling that I see teams using to manage their relational databases is crap.</li>
Having to handle live data makes refactoring harder (but not impossible).</li>
SQL is a language from the stone-age of software.</li>
</ul>
But that's no excuse for not doing the best you can for the problem at hand.</p>
There are answers to these problems</h2>
Some of the answer is just accepting that it's harder and learning to be effective. For example I find VsVim</a> and SQL Prompt</a> to be great for editing SQL.</p>
Redgate</a> have a great suite of tools for managing the full lifecycle of a database. Use them. I particularly like the way ReadyRoll</a> handles migrations. RedGate tooling has excellent integrations with Octopus deploy</a>.</p>
So if you're in the situation I've seen with a database you virtually can't change and nothing but a visual studio database project in place that may or may not be in place with production, then try the following:</p>

Throw away the database project.</li>
Set up ReadyRoll in your visual studio project.</li>
Grab your production schema, and use that as a base in your ReadyRoll migrations. (It has a neat feature for a first "migration" that will only be run on a new database build, so it can build you a dev database from scratch, but it won't attempt to recreate production).</li>
Configure ready-roll to use branch folders.</li>
Start creating migrations in feature branches.</li>
Set up CI / test / QA / pre-prod environments (or whatever you call them in your team) to drop and rebuild the database with every build. You can make this faster by using SQL Server snapshots.</li>
Each environment then runs the database migrations that match the version of the codebase that was released to that environment.</li>
Push changes up through the environments until those migration scripts have been run so many times you are no longer afraid to run them.</li>
</ol>
Now you can refactor your database with confidence just like you can with the code, and it can stop being something you pretend isn't there and start being something you are proud to show off.</p>
There are many potential complications, such as re-indexing tables with lots of data in production, but these are not things that an intelligent well-functioning DevOps team can't handle well once the basic process is in place.</p>
Further Reading</h2>

https://gist.github.com/timabell/6fbd85431925b5724d2f</a> - source control your schema documentation (ms_description attributes)</li>
https://dba.stackexchange.com/questions/515/how-do-you-document-your-databases</a></li>
</ul>

If you like this, you might appreciate the tool I'm working on to shed further light on the dark corners of your SQL database, check it out at http://schemaexplorer.io/</a> and be sure to sign up to the waiting list.</p>


My approach to my work
2018-03-10T00:00:00+00:00
I've realised that something that sets me apart from other developers with similar buzzwords on their CV is the way I think about things and the principles I think are important. But I've never put them on my CV or tried to express them in writing before. I'm not perfect and may fall short sometimes but these are the perfection I strive for.</p>
Tell the Truth, Even if it's Hard</h2>
I don't believe in selling good news only and robbing people I work with of the negative information that could be critical to making good decisions. E.g is this project going as well as expected?</p>

No. - opportunity to take corrective action by those with the authority to make sweeping changes</li>
Yes (a lie) - temporary happiness followed by reality catching up and being in an even worse situation</li>
</ul>
Great episode on the matter: The Startup Chat with Steli and Hiten: 288: Startup Lies https://thestartupchat.com/ep288/</a></p>
Strong Opinions, Weakly Held</h2>
Which means I have strong opinions about what the right thing to do is based on my experience and the information available at the time, but that if I am presented with new information then I am open minded to changing my opinion. I first heard about this while on contract with GDS, an organisation with an amazing culture for doing the right thing and getting things done. Here's a small example: I spent many many years preferring the C# style of skipping curly-braces for one-line if</code> blocks, based on the understanding that the less boiler-plate there is the easier it is to read the code. And then a fellow dev argued that it was better to always include them for the sake of reliability. I was initially unconvinced, but I always appreciate hearing a contrarian view and I proceeded to research extensively the pros and cons. Given the constant problems with indentation (spaces/tabs) in many projects that can make an if block look like something it isn't if you aren't paying close attention, and the automatic merge present in many source control systems, I was persuaded to change my view and will now argue just as strongly for the new stance</a>. For an excellent example of a serious security problem that might have been avoided by always having braces, read the "explanation of Apple's goto fail SSL bug</a>", a coding error which has the hallmarks of an unfortunate choice made by a merge algorithm.</p>
Communicate to be understood</h2>
I've had feedback that my communication with stakeholders is good, and that I can explain complex technical issues clear to less technical people. I think it's important that those around a project such as senior stakeholders, users, related support people and collaborating suppliers all have the best understanding of the state of a project and any issues. Without clear information in a language they are comfortable with then they are left unable to help the project succeed. I think it's important not to dumb things down as that results in losing important details, but instead take the time to help people understand the details they need to and why they might be important. As the technical experts it's up to us to make sure that others have the knowledge they need to do their jobs the best they can.</p>
Help others</h2>
I've also had feedback that one of my strengths is mentoring and sharing knowledge with other team members, technical and non-technical. It's certainly true that I do love helping others improve their own skills and knowledge where I can, and sharing what I've learnt over the years (ahem, decades). This is partly just because I love what we can do with technology and want everyone to gain the maximum benefit from it.</p>
Continuous Personal Improvement</h2>
I read the 7 habits of highly effective people</a> by Covey many years ago, and it is truly life changing because it changes your thinking from fatalistic "this is how I am" to "I can become a better person", and gives you some great mental tools for doing so. I have practised this consciously ever since. I particularly liked the quote I heard recently "What would you tell your younger self? Nothing, they wouldn't be able to understand what I know now, I'm a different person</em>" (or words to that effect).</p>

</a>
</div>
Follow Programming Best Practice</h2>
Our industry has learnt a lot over the last 50-odd years about what's effective. A lot of best practice has been captured in books, internet articles, and importantly libraries and frameworks. Strive to stand on the shoulders of the giants.</p>
Elegance, Practicality, and Definitely No Dogmatism</h2>
It's easy to get drawn in to the latest hip thing and apply it to everything you do no matter how inappropriate. Every situation should be assessed for the trade-offs that will be appropriate. One of the challenges of defining best practice in programming is the variety of problems it is used to solve. Should we use the same rules to produce a life-support system with no ability to push updates out as we should for a startup SaaS with no users yet? Clearly not, yet you'd wonder given some of the views presented; usually of the form "This worked on my project so you MUST now use it in ALL YOUR CODE!!!</em>" (This is the way Test Driven Development was introduced to me, and eventually I realised this is probably because it came largely from the dynamic language Ruby that lacks all the compile time checks I'm used to in C#, and as such complex systems have more failure modes as the codebase changes over time, resulting in a greater value of extensive unit testing.)</p>
Emergent Architecture</h2>
Planning is good, don't get me wrong. But don't restrict your options until you need to. The idea of emergent architecture is do the simplest thing that will work (following the You Aren't Going to Need It principle, aka YAGNI), and then as it becomes apparent you have a need for a particular abstraction or pattern that's when you apply it, through the magic of refactoring. I've seen too many projects that have been made many times more expensive than necessary because at an early point in the project a developer who had previously been burned by a lack of some structure or other made damn sure it was in their next project, regardless of whether it was actually necessary for the problem at hand. Another classic is learning of a pattern and applying it without it actually being needed. A common example of this I've seen is adding a "repository" layer to a codebase on top of entity framework, not realizing that entity framework already implements the repository pattern</a> (along with the unit-of-work pattern). I spent years learning about the various approaches to the architecture of software projects. Initially I thought I need to apply all of this best-practice to every project, but was troubled by how long that would take and found myself guiltily doing just enough to get the job done.</p>
Technical Debt and Refactoring</h2>
Refactor problematic things as they cause you problems. Simple. I really can't say it better than this: https://ronjeffries.com/xprog/articles/refactoring-not-on-the-backlog/</a></p>
The Agile Thing</h2>
I learned about eXtreme Programming (XP), scrum & kanban (I've actually worked in a real factory that used kanban). I've worked in many teams at various levels of competence on the whole "agile" thing and to me it seems to be down to good people making the best use of the available ideas for the problems at hand. Got no process at all, or something horribly like a waterfall? Add some elements of scrum and get better. Nailed scrum and feeling constrained? Do some more reading up, decide where to go next. I don't pretend to be an expert in "agile", but I love working in effective teams making use of the best methods available, and I love helping teams improve where I can. I'm not particularly interested in climbing the management-track ladder, I'm happy to work as a member of great teams. You can't do it all, and I don't want to get too far from my coding skills.</p>
Resources</h2>
These are things I've read that have guided my thinking towards where it is today (along with my personal experiences).</p>
Books</h3>

Mythical Man Month</a> - a timeless classic</li>
</ul>
Refactoring and technical debt</h3>

https://ronjeffries.com/xprog/articles/refactoring-not-on-the-backlog/</a></li>
http://laughingmeme.org/2016/01/10/towards-an-understanding-of-technical-debt/</a></li>
</ul>
Compassion</h3>

https://jml.io/pages/your-code-sucks-and-i-hate-you.html</a></li>
</ul>
Technical details</h3>

http://endoflineblog.com/gitflow-considered-harmful/</a></li>
</ul>
New thinking</h3>
Things that I've recently come across that I wish to share / promote</p>

https://simpleprogrammer.com/agile-is-dead-code-review/</a></li>
</ul>

Get In Touch</h2>
If you need someone who you can rely on to bring more than just a list of keywords to your project then get in touch: tim@timwise.co.uk</a></p>


Contract developer to entrepreneur - resources
2018-02-16T22:31:38+00:00
I'm not there yet so I'm no authority, but here's some things that I think are fab resources on my journey. They are helping me change my thought patterns and behaviour to make success more likely.</p>
Books (roughly in order of my journey)</h2>

The e-myth (revisited)</a></li>
A better mousetrap - the business of invention</a></li>
The Art of the Start</a></li>
Key person of influence</a></li>
So good they can't ignore you</a></li>
</ul>
More books over on my goodreads profile</a>.</p>
Some more suggestions on the cocoon blog</a></p>
Inspiration</h2>

https://www.softwarebyrob.com/2015/03/26/the-stairstep-approach-to-bootstrapping/</a></li>
</ul>
Podcasts</h2>

Bootstrapped Web [Website</a>] [Feed</a>]</li>
Marketing for Developers [Website</a>] [Feed</a>]</li>
Noah Kagan Presents [Website</a>] [Feed</a>]</li>
Rogue Startups Podcast [Website</a>] [Feed</a>]</li>
Startup Chat [Website</a>] [Feed</a>]</li>
Startups For the Rest of Us [Website</a>] [Feed</a>]</li>
The Art of Product [Website</a>] [Feed</a>]</li>
The Indie Hackers Podcast: How Developers are Bootstrapping, Marketing, and Growing Their Online Businesses [Website</a>] [Feed</a>]</li>
The Startup Chat with Steli and Hiten [Website</a>] [Feed</a>]</li>
The Tim Ferriss Show [Website</a>] [Feed</a>]</li>
Tropical MBA - Entrepreneurship, Travel, and Lifestyle [Website</a>] [Feed</a>]</li>
Zen Founder: Startup. Family. Life. [Website</a>] [Feed</a>]</li>
</ol>
My feed list</a> exported from AntennaPod</p>
Products</h2>

github-pages</a> - for hosting landing pages etc</li>
logojoy.com</a> - $65 automated and very good logo generator</li>
godaddy - for domains</li>
</ul>
Education</h2>

http://tinymarketingwins.com/</a> - learn marketing even before you have anything to sell</li>
https://designacademy.io/</a> - design for developers</li>
https://www.ditchinghourly.com/</a></li>
http://startupclass.samaltman.com/</a> - also on youtube & as a podcast:</li>
</ul>
https://www.youtube.com/watch?v=CBYhVcO4WgI&list=PL5q_lef6zVkaTY_cT1k7qFNF2TidHCe-1</a></p>
Community</h2>

Indie Hackers</a></li>
</ul>
People</h2>

Noah Kagan</a></li>
Tim Ferris</a></li>
Courtland (Indie Hackers)</li>
Laura Elizabeth</a></li>
Jonathan Stark (Ditching Hourly)</li>
</ul>


Down with SDV! long live SSE!
2017-12-29T02:59:54+00:00
I can't keep up with myself on this startup journey, but I hope I can give you a peek into my journey as I go. I've had couple of revelations recently which I'll explain here:</p>
Discovering your needs</h2>
I've managed to talk to more people about what problems I could solve and have started to learn more about what value this tool might provide (solution-to-problem instead of the usual solution-looking-for-problem).</p>
I had originally thought of this tool as something for viewing data, but I've had several people express an interest in the ability to visualize relationships. This was surprising to me as personally I've been using SqlHawk</a> (my fork of SchemaSpy</a>) to get to grips with database relationships, but I hadn't considered that they don't get widespread use in the circles I move in; presumably due to a mix of awareness and what a pain they are to set up (Java, shudder).</p>
Having explained my vision for the tool to my friend and ex-colleague Ben, I managed to ask him to say in his own words how he might hypothetically describe it to others, and he uttered the magical words: "well it's kind of a schema explorer". And suddenly I understood that the real value of the tool I'm creating is not in the ability to poke around the data per-se, it's actually the ability to come to an unfamiliar relational database and be able to much more quickly understand how it hangs together. (It'll have value in familiar databases too, but I like this focus, it feels higher-value.)</p>
As a result "diagrams" is now the top-priority feature being added to SDV.</p>
It also gives me a new name for Sql Data Viewer...</p>

</a>
</div>
New name, new domain</h2>
The product has been renamed to Sql Schema Explorer</strong>, which can now be found on the shiny new domain schemaexplorer.io</a></p>
For the new website:</p>

I've given up on all the fancy wordpress designs and hosting</a> I was using for the old domain and have gone for github pages</a>.</li>
All the bloated css and javascript is gone, leaving nothing but a hand-crafted flat html</a> file that should load beautifully on any connection thanks to being under 1kb of content in total with no trackers or external resources.</li>
It should reliably render on most devices with minimal CPU load or delay thanks to the lack of bloated and fragile javascript / device detection etc.</li>
I've ditched the fancy drip sign-up form in favour of a simple email link (why solve the volume problem when I don't have it yet?).</li>
I've ditched the google analytics tracker because it's a vanity metric if it's not money in the bank; and these trackers have hidden costs (page load time, leaking your customer's data to google etc). If I get a sudden influx I can just ask them how they found me rather than reading between the digital lines.</li>
I've stripped the content back to almost the bare minimum to try and drive focus towards email signups so that I can then engage in-depth through that. I need enough that I can point people at it in conversation to get them signed up. I only need a dozen early customers to help me shape the product so I don't need to optimize for complete strangers who've never heard of me yet.</li>
I've created my own personal style of  design that favours minimalistic css and markup to keep page load fast and complexity down whilst still giving structure to the message and a little bit of easiness on the eye (e.g. off-white/black, read the css if you like). I'll be continuing to improve my design skills</a> as I work on this project, though I'm not against outsourcing when I need to.</li>
</ul>
Money, money, money</h2>
You may remember I wrote about making SDV free</a> not too long ago. With the above new understanding about the value that the product can provide and what it will take to build a full-featured product for that market I am no longer concerned about being immediately copied, and think it has sufficient value to be a viable product in its own right. As such I will be charging for it after all.</p>
I'll be following the great advice from the episode of Startup Chat - 268: Encore Episode – How to Get Your First 10 Customers</a> that I heard recently (highly recommend this if you're also thinking of getting a startup / side-project off the ground).</p>
The starting price point will be £150 per single user/machine. This is based on the "3x the first number you think of" formula because I think  they are right that first instinct is off; after-all I have to make this a viable business otherwise development will stop and no-one wins.</p>
I'll be offering the recommended money-back guarantee for my first customers; that way I can prove that there is real value in what I'm creating whilst maintaining my high ethical standards for fairness in business.</p>
Still here?</h2>
Great! Thanks so much for reading this. If you're interested in keeping up to date or being involved in the early life of this new product then head over to schemaexplorer.io</a> and sign up!</p>


From contractor to entrepreneur - motivations
2017-11-19T23:01:08+00:00
I'm not sure what changed but I used to be happy publishing open source code on github and thinking I'd done the world a mighty favour. But slowly I started to become uneasy. How much good have I really done if I publish something that only a couple of people find, let alone benefit from? I also started to find it harder to find any time to spend on software to give away for free in between the day job and my growing family. I think that might be why I now find myself pursuing for-profit side projects. It's not everything but revenue is some measure of the value I am creating for others. Revenue is also a way of spending more time creating that value; every pound I earn from my creations is a pound I don't have to earn from the day job, and because I'm a contractor I have the luxury of being able to adjust my work levels. If I achieve any success with my own projects that would allow me to invest more of my time in them and generate yet more value without sacrificing family life.</p>
Ever since I entered the world of work I've wanted to create things that help people in some way. I think one of the reasons I still love software development is the joy of watching someone realise I've made one small piece of their life or job better, easier or faster in some meaningful way. Creating my own products that gain wide adoption is the ultimate expression of that for me.</p>
I've already had some fantastic unexpected benefits of starting on this journey. It's reminded me to be humble, and given me even greater respect for anyone who has created a business. It's pushed me to read and listen to many things I wouldn't have thought of before, and reading with a goal in mind seems to be a much deeper learning experience. It's enabled me to have fascinating conversations with people I otherwise wouldn't have had a reason to talk to, such as the lovely generous people at grow@greenpark.</p>
I've discovered huge holes in my skills that I was oblivious to in my cosy enterprise-developer role, such as marketing, and I am now super-motivated to learn more.</p>
Maybe I can build something that you would find useful? What's the biggest hole in your digital life?</p>
Please sign up for my mailing list</a> so we can follow this journey together.</p>


Looking for beta-testers for my new Free Sql Data Viewer tool
2017-11-18T01:28:08+00:00
Hi, my name's Tim and I'm a UK based developer trying to build my first startup.  I'm working on a data-browsing tool for Sql Server etc. that I really believe can help people do their jobs.</p>
I'm reaching out to the online communities to try and find people who will find it useful and who are prepared to help me shape the early product.</p>
Please take a look at my offering on https://sqldataviewer.com/</a> ... and if you like the look of it then please sign up to my waiting list</a>.</p>
Thanks for your precious time!</p>
Tim Abell</a></p>


Why SDV could be a billion dollar business
2017-11-17T23:10:48+00:00
Okay maybe a billion is a long shot, but I read the Hiten Shah article on how Trello could have been a $1B business</a> (via startup chat</a>) and was surprised to discover there are lessons for how I can turn Sql Data Viewer</a> into a sustainable business.</p>
What trello and SDV have in common:</p>

Mass (horizontal) appeal</li>
Easily copied</li>
Difficult to sell premium versions of the general purpose product ("stickers" anyone?)</li>
</ul>
I was going to just sell SDV for £100 and be done with it, but this article showed me a better way.</p>

Give the general purpose tool away for free (as Trello does)</li>
Collect contact information in exchange for the value provided by the free tool.</li>
Use the contact list to have conversations with businesses that use it in order to...</li>
Find out how integrating tightly with their business could provide real value over and above the general purpose tool.</li>
Build vertical-specific integrations, stickyness & marketing.</li>
Profit! (No underpants required. Or gnomes.)</li>
</ol>
What do you think? Drop me a line: tim@timwise.co.uk</a></p>


Git for TFS users
2017-10-27T20:44:06+01:00
I'm considering creating a 30 minute video to help teams transition from Team Foundation Server source control to git (hosted with TFS or Visual Studio Team Services - aka VSTS).</p>
Please register your interest</a>. If I get enough interest in this then I'll do the work to put it together and launch it.</p>
![tfs-to-git-terminal]({{ site.baseurl }}/assets/tfs-to-git-terminal.png)</p>


Choosing wordpress hosting for a new idea
2017-10-22T11:32:08+01:00
I need somewhere to keep the marketing site/content for my new Sql Data Viewer</a> product. (Since renamed to SQL Schema Explorer</a>.</p>
I could use github pages</a> or something similar, but I've found the overhead of needing a dev environment to make any change puts me off getting things done. I've been using wordpress.com for this blog and am pretty happy, especially having a mobile app. I've also heard using wordpress makes it easier to outsource content/design if you have enough success to make that worthwhile.</p>
I don't fancy maintenance, security patching and backup, so the marketing site is currently on wordpress.com</a></strong> where I've paid for the "personal" plan at £3/month</strong> to remove ads and the £0.92/month</strong> for mapping a domain. Affordable for contactor running a shot-in-the-dark startup side-project that may never make a penny. Note that you only get their built in analytics, you can't add google analytics, and you can't install plugins on that plan.</p>

To get google analytics you need "premium" at £7/month</li>
To be able to add plugins you need "business" at £20/month</li>
</ul>
This is on top of the £1.15/month</strong> for domain registration and £10/month</strong> for a VPS (virtual private server) to run my demo site from, both from the excellent bytemark</a> hosting company, who've been fab on support from day one.</p>
Just for completeness: the demo site connects to sql azure instances are easily within the $40/month included with my MSDN Professional subscription that I use for contracting. Handy.</p>
I've been listening to lots of startup podcasts (email me</a> if you want the list), and have discovered the mighty Justin Jackson</a> and his excellent Marketing for Developers</a> content - I recommend you buy it if you're a developer thinking of selling a product / service. I'm now working through trying to apply the lessons in the book to my own product. I got to the analytics bit and hit a wall. Wordpress.com (the wordpress hosting site, not the software itself) force you to upgrade to the business plan at £20.83/month</strong>. Given I don't know how long this is going to take me I'm not keen to start ramping up costs to that extent with no obvious route to a return on the investment.</p>
I've looked at the three recommended in Justin's book (plus more added more recently), and the starting prices are:</p>

Siteground</a>: Startup plan (like it!) £2.75/month</strong> - looks like this might be a winner then! Oh wait, that's a sign-up discount, it's actually £6.95/month</strong>. Hrumph.</li>
Krystal</a>: Personal £18/month</strong></li>
WPEngine</a>: Personal $29/month - about £22/month</strong>.</li>
Pagely</a>: VPS-1 $499/month - about £380/month</strong> - ouch! I don't think I'm their target market. I wonder if they repriced, perhaps Justin should drop this one from the book.</li>
</ul>
And that's where I'm at right now. Maybe I'll just shove a docker wordpress image on the VPS, it's already a docker host. I need to figure out hosting multiple sites on one IP for the demos anyway. But then again, I don't want to be a wordpress sys-admin. Or maybe I'll rethink the static site thing.</p>


Home server docker-compose
2017-08-29T21:26:28+01:00
Another step in making a throw-away home server: docker-compose.</p>
Find my compose file and setup script at https://github.com/timabell/home-server-docker-compose</a></p>
I learned how to use docker-compose while working on reverse-proxying sdv, find out more: https://github.com/timabell/sdv-docker</a></p>


Home server backups
2017-08-29T20:31:08+01:00
The setup</p>

Ubuntu server domU xen host

Ubuntu server xen VM with LUKS full disk encryption

docker-compose

syncthing with built-in "Staggered File Versioning"</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
Plan</p>

Plug in usb external hdd, with full disk encryption (i.e. a LUKS partition taking up almost all the space, plus a little fat32 with a text file in case anyone finds it and wants to return it).</li>
Use LUKS key chaining to be able to unlock the disk without entering a password</li>
Use autofs to automatically mount/unmount so that it's safe to unplug when a backup isn't running.</li>
Use rsync to push all the files from the syncthing data directory onto the usb disk.</li>
</ul>
Somehow need to make the disk accessible to the VM.</p>
Finding the luks partition when plugged into the host:</p>
tim@spot:~$ sudo fdisk -l</span></span>
...</span></span>
Disk /dev/sdc: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors</span></span>
Units: sectors of 1 * 512 = 512 bytes</span></span>
Sector size (logical/physical): 512 bytes / 512 bytes</span></span>
I/O size (minimum/optimal): 512 bytes / 512 bytes</span></span>
Disklabel type: dos</span></span>
Disk identifier: 0x6380ad37</span></span>
</span>
Device Boot Start End Sectors Size Id Type</span></span>
/dev/sdc1 63 21579 21517 10.5M c W95 FAT32 (LBA)</span></span>
**/dev/sdc2 21580 1953147527 1953125948 931.3G 83 Linux**</span></span>
</span>
tim@spot:~$ ll /dev/disk/by-uuid/</span></span>
...</span></span>
lrwxrwxrwx 1 root root 10 Aug 29 21:51 **6ca09b72-8c9b-4571-8943-9f1d520671ab -> ../../sdc2**</span></span></code></pre>
Confirm it's the luks partition:</p>
tim@spot:~$ sudo cryptsetup luksDump /dev/disk/by-uuid/6ca09b72-8c9b-4571-8943-9f1d520671ab</span></span>
LUKS header information for /dev/disk/by-uuid/6ca09b72-8c9b-4571-8943-9f1d520671abVersion: 1</span></span>
Cipher name: aes</span></span>
Cipher mode: xts-plain64</span></span>
Hash spec: sha1</span></span>
...</span></span></code></pre>
I'll update this post with any details as I progress. Don't hold your breath though!</p>


Running sdv in docker
2017-08-13T20:35:14+01:00
Just bought a shared server with bytemark</a>. (£10/month), Installed vanilla ubuntu 16.04LTS server using the control panel (virtually one-click).</p>
![Servers - Bytemark Manager - Mozilla Firefox_143]({{ site.baseurl }}/assets/servers-bytemark-manager-mozilla-firefox_143.png)</p>
(Love the cancel button text!)</p>
Ssh'd in, created my own user to use with sudo instead of root. Ran the following, and immediately had a copy of sdv listening on the internet.</p>
sudo apt install tmux docker docker-compose</span></span>
sudo adduser tim docker</span></span>
# logout & reconnect to get new group to take effect</span></span>
wget https://raw.githubusercontent.com/timabell/sdv-docker/master/docker-compose.yml</span></span></code></pre>
and then for the magic:</p>
tim@sdvweb:~$ docker-compose up</span></span>
Pulling sdv (timabell/sdv:latest)...</span></span>
latest: Pulling from timabell/sdv</span></span>
d5c6f90da05d: Pull complete</span></span>
1300883d87d5: Pull complete</span></span>
c220aa3cfc1b: Pull complete</span></span>
2e9398f099dc: Pull complete</span></span>
dc27a084064f: Pull complete</span></span>
eb1a4736b68c: Pull complete</span></span>
0706cf350247: Pull complete</span></span>
1d0ac78e96a5: Pull complete</span></span>
Digest: sha256:9003a79f019b3ee16e7c6324afd275b4535f867602e63db317e430528e2a6771</span></span>
Status: Downloaded newer image for timabell/sdv:latest</span></span>
Creating tim_sdv_1</span></span>
Attaching to tim_sdv_1</span></span>
sdv_1 | ./sdv-linux-x64 -listenOn 0.0.0.0 -port 8080 -driver sqlite -db /data/Chinook_Sqlite_AutoIncrementPKs.sqlite</span></span>
sdv_1 | 2017/08/13 19:14:44 Sql Data Viewer v0.4; Copyright 2015-2017 Tim Abell <sdv@timwise.co.uk></span></span>
sdv_1 | 2017/08/13 19:14:44 ## This pre-release software will expire on: 2017-10-01 00:00:00 +0000 UTC, contact sdv@timwise.co.uk for a license. ##</span></span>
sdv_1 | 2017/08/13 19:14:44 Starting server on http://0.0.0.0:8080/ - Press Ctrl-C to kill server.</span></span></code></pre>
And that's it, listening on the internet!</p>
![Sql Data Viewer - Mozilla Firefox_144]({{ site.baseurl }}/assets/sql-data-viewer-mozilla-firefox_144.png)</p>
Any changes I make will be to the docker and compose files</a>, resulting in trivially easy to repeat deployments, making the server completely throwaway. Huzzah! Docker is awesome.</p>
I've also bought a domain for the product, but haven't set much up on it yet: http://www.sqldataviewer.com/</a></p>
Next up, getting nginx reverse proxy to provide ssl. And finishing the refactor I was in the middle of to get data types to behave, and finishing the automated regression tests I'd just started (needed for reliable multi rdbms support), and finding my market, and marketing the product, and doing more features, and build the marketing automation to drive sales, etc etc. Not too much to do then! Your support would be appreciated, if you're interested make sure you sign up to the mailing list now</a>!</p>
If you're wondering why docker matters more generally then listen to this podcast episode: Hanselminutes: Practical Containers for Developers with Aja Hammerly http://www.hanselminutes.com/default.aspx?ShowID=18514</a></p>


Patreon open source funding
2017-06-03T10:24:40+01:00
I'm now on patreon</a>.</p>
Hopefully this will provide enough funding that I can take better care of the open source projects that I look after. Since becoming a daddy it's become hard to find the time to do free work on these projects even though I love working on them. Combine that with the ambition to create a startup which eats up the rest of my spare time and the result is that these projects that I love are getting stale.</p>
As a contractor my time is available for purchase. Unlike in permanent employment I can vary the amount of paid client work I take on in a year, so if I get serious funding for the open source work I can legitimately scale back my normal client work to make time for it. I'm not looking for people to buy me a pint out of gratitude here (though I wouldn't complain!), I'm looking for serious funding from people and organisations that benefit from all the hard work put in so that we can all benefit from further improvements to the great public codebase that is open source.</p>
As an example, my ef-enum-lookup nuget package</a> has had 18k downloads to date, and I suspect this is unlikely to be used in hobby projects given the nature of EF (entity framework, an object-relational-mapper from Microsoft). So a fair number of commercial operations are benefiting from the improvements to their database that this project brings. I've had several requests and even some pull requests for improvements and features that I would dearly like to add, and that these businesses would benefit from; if they pooled money via Patreon then it would actually be an economical way for them all to benefit from the improvements. For example I counted six different individuals expressing an interest in custom schema support</a>. What if they or their companies all contributed?</p>
If you want a project to not dissolve into a mess then even pull requests are not free. Although there is a probably-working improvement, it may not fit the style of the project, it may introduce other issues, it may be missing unit tests, it may be written by a less experienced programmer and need additional work and/or support.</p>
I've had to choose monthly or per-feature on Patreon, and I've gone for monthly because there's more to an open source project than just bug-fixes and features - you also have to package it, unit test it, host it, refactor it, document it, set up continuous deployment or whatever the flavour of the day is, and respond to people who are having issues or who have contributions to offer. It seems to be a better match to the nature of open source development to ask for ongoing support as once someone is using a project they are likely going to benefit from using it for some time, and even a new feature will have a long life providing value to users.</p>
I haven't picked a specific daily rate I'd work on this for yet, but I'd probably give some discount over my normal contract rates just because I think it's worthwhile and I enjoy it.</p>
Do you or your employer use any of my open source projects? Fund me to make them better! Head over to https://www.patreon.com/timabell</a> with your boss's credit card and then drop me a line to let me know what's of interest to you.</p>


Dustbin of ideas - IT Contractor Buddy
2017-05-12T12:32:07+01:00
I've talked to a couple of people experienced in the recruitment industry and it appears that the contractor buddy</a> business idea has some fatal flaws and is unlikely to succeed.</p>
You might be surprised to hear that I consider this a success on my journey to startup success.</p>

I've had a business idea (not an easy thing to do!) which is more realistic than many of my previous ideas showing that my ability to spot such opportunities is improving. (It has a clear revenue model and its fairly easy to see how it could be built.)</li>
I've not become too attached to it to hear valid criticism.</li>
By putting it out there and discussing it openly with anyone willing to help me out I've rapidly found the flaws in it without wasting valuable time or money. (Imaging if I spent two years building it to then have the same flaws sink it.)</li>
I've managed to find contacts from a broad range of backgrounds who have helped me to crystallize my idea - this is entirely thanks to spending time at the fabulous co-working and startup community space grow@greenpark</a>.</li>
</ul>
So what sank it?</p>

Even though the company would not be inserted into the financial transaction, they would still need to be a communication relay between employer/agent and candidate, and the received opinion is that this just won't fly.</li>
The number of contractors who could find this useful could be small, with the highly in-demand people not having to put in much work to get the next contract (therefore not gaining much value from the service), and the the not-in-demand contractors are not a market the business would want to be primarily associated with as it could become a by-word for low-quality talent. This leaves the middle ground which may not be enough of a market to support the business. Before I'd talked to anyone I'd assumed that all contractors would want this (based on sample size of one - me).</li>
</ul>

</a>
</div>
The investment required to prove/disprove these suppositions would be significant, as I think you'd basically have to build the functioning business to see what happens in reality. So on that basis, "I'm out!"</p>
So what now?</p>
Well there's my Sql Data Viewer</a> that I haven't given up on (check it out, let me know what you think and join the mailing list).</p>
There's also the possibility of a pivot</a>. Maybe there's room for something to help contractors without getting in the middle? Maybe it could send mail using their real email address and handle responses? Maybe there's already something like that out there and I should just use it instead of trying to invent my own! Let me know in the comments if you know of anything or already have ways to ease the pain.</p>
Thanks for listening.</p>


IT Contractor buddy
2017-05-03T13:31:51+01:00
A new job-hunting assistant for contractors</h2>
Calling all IT contractors; would you pay to have some of the pain of contract hunting taken away?</p>
I'm considering launching a business for IT contractors like myself that will do the tedious and time-consuming bits of job hunting.</p>
If this is something you think you'd be interested in then sign up to the mailing list for early access</a>.</p>
Concept</h2>

Search job postings for you, letting you know when good matches come up, across multiple job boards.</li>
Submit your CV on your behalf</li>
Keep track your applications (when you sent your CV where for what rate).</li>
Handle first contact from companies / recruiters, validating it's worth your time</li>
A simple monthly fee, with no long-term commitment.</li>
No cut of your contract rate and no fee per placement.</li>
</ul>
A concierge service for IT contractors.</p>
As I know from first-hand experience the job hunt is a time-consuming slog where being on top of all your leads is critical to avoid being taken advantage of rate-wise by a sharp-eyed contingency recruiter, or having a deal ruined by duplicate CV submissions.</p>

</a>
</div>
How is this different from all the other recruiters?</h2>
In house recruiters are agents of the hiring company, and you may have to deal with many of these. We are working for you, the contractor, no matter how many companies and agencies you are talking to. They can only help you with respect to their company.</p>
Contingency recruiters (i.e. 3rd party recruiters) are trying to insert themselves into the deal between you and a client, sometimes with the okay of that client, sometimes without. Unlike them we are not trying to skim money off your contracts, instead we are providing a service that helps you avoid the painful bits of the job hunt, for a straight-forward monthly fee that reflects the value we provide to you. Value for value. These recruiters will not help you with leads from other recruiters, in fact letting them know about other leads is a recipe for disaster.</p>
So neither of these will help you with your overall job hunt. You still have to manage all your leads yourself.</p>
How is this different from a talent agent?</h2>
You may have read the new-yorker piece about</a> the agent 10x</a>.</p>

They charge 15% of your rate</a> (a pretty big cut imho!) whereas we charge a flat rate for the service we provide; value for value.</li>
They are limiting themselves to the mythical 10x'er. So even though they offer a lot of the concierge I'm offering, you probably can't get them to take you on. We however will take on even non-rocket-scientist contractors as we scale, free from BS hype about "rockstars".</li>
Our service is clearly working for you and no-one else whereas with 10x's  percentage fee it's not clear who they work for no matter what the marketing says.</li>
10x are attempting to find clients directly, just like every other contingency recruiter. We don't mind where your work comes from as we aren't trying to insert ourselves into the deal, so you won't be limiting your market by working with us, in fact we'll make it easier for you to cover more of the market by doing the hard work for you.</li>
</ul>
Sign up now!</h2>
If you think this is something you might use, then please sign up to the mailing list</a> so you can get first access. There will initially be limited spaces for clients so that we don't overstretch, so sign up now</a> for first-come first-served early access.</p>


New home server with Xen and docker
2017-03-10T00:03:48+00:00
No cloud mkay?</h2>
Going against the current received wisdom I personally am not keen to trade the convenience of the public cloud sync services such as dropbox, onedrive, google drive etc for the fact that this means that all my files traverse the public internet and live on someone else's metal. Sure it might be encrypted, but what if my traffic is intercepted and stored, and then that encryption is later found to be flawed?</p>
I'm probably just as likely to get pwned running my own setup, but hey at least I get to learn something along the way.</p>
Another thought is that regardless of location - home, office, cloud, vps host or data centre - it's all just (server) software. So why shouldn't we be able to get the amazing seamless experience when self hosting? It seems to me that it comes down to economics. Cloud hosting gives an opportunity for charging directly or worse using underhand tactics like owning your data or selling you as a product to advertisers. This means there is more funding for cloud systems, and therefore more developers are working on it. Even ethically minded software developers have to eat and live somewhere. I think just like Linux vs Windows we will get a perfectly good open source solution, but it will likely lag behind.</p>
Disk encryption</h2>
I have a new machine to use as a server so have an opportunity for a fresh approach. I want my server encrypted at rest in case the whole machine gets stolen from the house to keep the scoundrels from having easy access to all my files.</p>
The previous machine I installed with encrypted LVM as available with Ubuntu server's installer; but this means going to the machine to unlock it after every reboot / power failure / kernel upgrade. I did have remote unlock via dropbear ssh</a> set up but it's hacky to set up and non-trivial to use.</p>
This time I've gone for a different approach:</p>

An unencrypted install of ubuntu server (DomU) with an ssh server, so no password required to boot up, which does nothing more than give access to:</li>
A Xen VM install of Ubuntu server (Dom1), this one with LVM disk encryption. This is where all my files etc will live. Installed with virt-manager</a>, which is a very convenient UI for initial setup.</li>
</ul>
So now after a reboot I can ssh -X</code> into the DomU server, then using virt-manager I can see the Dom1 VM waiting for the disk encryption key and provide the passphrase.</p>
Xen</h2>
Setup instructions for xen on Ubuntu</a>.</p>
It wasn't immediately obvious how to set up the networking. This was the network config that worked in the end (after ifdown/ifup, also tested with full reboot):</p>
$ cat /etc/network/interfaces</span></span>
source /etc/network/interfaces.d/*</span></span>
</span>
# The loopback network interface</span></span>
auto lo</span></span>
iface lo inet loopback</span></span>
</span>
auto xenbr0</span></span>
iface xenbr0 inet dhcp</span></span>
  bridge_ports enp1s0</span></span>
</span>
# The primary network interface</span></span>
auto enp1s0</span></span>
iface enp1s0 inet manual</span></span>
#iface enp1s0 inet dhcp</span></span></code></pre>Docker</h2>
Seeing as docker is well and truly coming to windows it's time I got better at this so, for this and also better isolation and security I'll try and set up as many of the services I want to run as I can through docker.</p>
First test case was transmission-bt</a> which I run to be a good citizen and help seed Linux Mint ISOs</a> using my A&A unmetered upload</a>.</p>
Searching for transmission on docker hub</a>, I didn't want the vpn one, I'm not pirating movies and my ISP doesn't do stupid filtering/shaping tricks on torrents. I chose the next most popular: https://hub.docker.com/r/linuxserver/transmission/</a></p>
To install and run it:</p>
$ sudo -i</span></span>
# useradd -r -s /sbin/nologin transmission</span></span>
# id transmission</span></span>
id=999(transmission) gid=999(transmission) groups=999(transmission)</span></span>
# cd /var</span></span>
# mkdir transmission</span></span>
# cd transmission</span></span>
# mkdir config downloads watch</span></span>
# chown transmission:transmission .</span></span>
# docker create --name=transmission \</span></span>
  -v /var/transmission/config/:/config \</span></span>
  -v /var/transmission/downloads/:/downloads \</span></span>
  -v /var/transmission/watch/:/watch \</span></span>
  -e PGID=999 -e PUID=999 -e TZ="Europe/London" \</span></span>
  -p 9091:9091 -p 51413:51413 -p 51413:51413/udp \</span></span>
  linuxserver/transmission</span></span>
# docker start transmission</span></span>
# docker logs -f transmission</span></span></code></pre>
useradd ref: http://askubuntu.com/questions/29359/how-to-add-user-without-home</a></p>
Once that was done I could see the transmission web UI at http://dom1vm:9091/</code>. Hurrah.</p>
syncthing</h2>
https://store.docker.com/community/images/linuxserver/syncthing</a></p>
Same deal with folders and users, then:</p>
# docker create --name=syncthing -v /var/syncthing/config/:/config -v /var/syncthing/data/:/data -e PGID=998 -e PUID=998 -p 8384:8384 -p 22000:22000 -p 21027:21027/udp linuxserver/syncthing</code></p>
Start on boot</h2>
docker update --restart=unless-stopped transmission</span></span>
docker update --restart=unless-stopped syncthing</span></span></code></pre>
http://stackoverflow.com/a/37479753/10245</a></p>
Todo</h2>

nextcloud</a> - has an iOS app that syncthing doesn't yet, might also have better selective sync

https://store.docker.com/community/images/wonderfall/nextcloud</a></li>
</ul>
</li>
plex server

https://www.plex.tv/</a></li>
</ul>
</li>
</ul>
Further reading</h2>

https://forum.level1techs.com/t/dexter-kanes-ultra-paranoid-encrypted-nas-completed/98340</a></li>
</ul>


New year new blog
2017-01-17T23:36:00+00:00
So-long blogger and thanks for all the fish.</p>
My content to date will live on here for as long as google keep the bits spinning.</p>
Find my new blog at http://blog.timwise.co.uk/2017/01/17/new-year-new-blog-happy-2017/</a> which will live for as long as I pay wordpress.com's bills and don't screw up my DNS config.</p>
See y'all on the other side. Don't forget to subscribe to the new RSS feed.</p>


Startup competitors - Data Viewer
2017-01-17T01:55:27+00:00
If you follow my ramblings you might know that I've revived my interest in
creating a commercial tool to help devs, dbas etc browse their sql databases.
I've done some research before and didn't find much on a par with what I have
in mind. I've done a bit more, and turned up a couple:</p>

A feature of sql
workbench</a> - I've seen the
product but didn't know it could do that. Not directly competition as it's
not nearly as smooth to use as what I'm building for this particular use
case.</li>
SqlSmash</a> - Something I'd not seen before, which
has fk based navigation (vid)</a>
that's basically what I'm building, albeit embedded in SSMS rather than
standalone.</li>
</ul>
This isn't the first time I've had the experience of getting started before
discovering another product that basically solves the problem I'm gunning for.
(Ready-roll</a> was
the last, since bought by red-gate).</p>
Now it's not that I thought my idea was magic or special or a unique little
flower, but why is it after working with SQL Server databases for 15 years that
I've only just noticed this tool (which I would have found useful and probably
would have paid for).</p>
If I created another would it be just as obscure? Should I still go ahead even
though it's a solved problem? Are there any ideas left on the earth?</p>
Something for me to consider this week. I might update this post with further
thoughts. Your thoughts are welcome.</p>


You might also like this post: Database tools I was surprised
existed</a></li>
I did end up building it, and it became https://schemaexplorer.io/</a></li>
</ul>


New year new blog, happy 2017
2017-01-17T01:00:09+00:00
I've used blogger for years, but the layout for code has not improved for a long time. I've also found my domain's website stagnating because of the hassle of updating it, so I've moved the whole kaboodle to wordpress.com</a>.</p>
Find my old blog and all its content at http://timwise.blogspot.co.uk/</a></p>
I'm not migrating the content from blogger because I think google will keep it online, and cool URIs don't change</a>. I've never been convinced of the value of people lugging their blog posts from one platform to the next, it seems to break more than it fixes most of the time. Like the one where the old host didn't need the trailing slash at the end of the URL and the new host did, breaking half the incoming links. It's only because I'm a web dev that I guessed what the fix was.</p>
Enough about that anyway. Happy new year :-)</p>
Please do add my new blog to your favourite blog reading software / rss reader etc., and do drop me a line on tim@timwise.co.uk</a> just so I know you're out there!</p>
The original of the header is a photo I took myself which you can find on flickr: https://www.flickr.com/photos/tim_abell/17111067690/</a></p>


javascript dates in firefox and locales
2016-09-27T07:31:00.002+00:00
Today I learnt:</p>
You actually have to reinstall firefox using a different download to get new Date(dateString)</code> to use a different date format (i.e. non-US).</p>
https://jqueryvalidation.org/date-method/</a></p>

This method should not be used, since it relies on the new Date</code>
constructor, which behaves very differently across browsers and locales. Use
dateISO</code> instead or one of the locale specific methods (in localizations/
and additional-methods.js).</p>
</blockquote>
So if you're using chocolatey or boxstarter like I am (my boxstarter
script</a>) you need to add
the locale flag:
https://chocolatey.org/packages/firefox</a></p>
choco install Firefox -packageParameters "l=en-GB"</code>.</p>
Reinstalling the package with --force</code> is sufficient to change it.</p>
It would seem chrome has no way change the format from en-US</code>.</p>
To be clear the correct solution is to do as the documentation says and not use
that method, however it's a bit mean having a pitfall like that in the API. And
sometimes you just have to work with the code you have...</p>


Yet another good-commit-messages post
2016-03-18T15:25:00+00:00
Why you should care about writing good commit messages</h2>

It is extremely likely you won't be the last person to touch this code.</li>
You will forget the details of exactly why you did something after a few weeks/months/years. Be kind to your future self.</li>
Source control systems out-last [access to] ticket trackers by many years (Jira, Github-projects, Trello boards, etc. etc., how many have you seen in your time working on code? A lot more than you've seen source control systems I'll wager.) A link to a ticket is not enough. Copy the relevant context from the ticket into the commit message.</li>
Not everything can be deduced from the code. Even with excellent variable/function/class/module names, and beautiful refactoring, the circumstances that let to a particular design or change are lost. Good comments adding context help, but sometimes you don't want to clutter code with temporally relevant comments; the source control gives you a suitable place to keep this information locked up with your patch of the day.</li>
When someone wants to know if they can delete a line of code in the future, if they can't fathom why it was put there then the only way to find out is to delete it and see what breaks. By adding context in a commit message they can use the source-control history to get that context, and then decide whether that context is still applicable.</li>
You might think your code is perfect and correct and needs no explanation beyond the code itself; but what if there's a bug? Now the only documentation is the buggy code. If I come to fix your code later how do I know what it was supposed to do without going back to first principles. What algorithm or design pattern were you trying to implement? What references did you use?</li>
Chesterton's Fence</a> - understand why something is the way it is before you change it.</li>
And finally, I have seen first hand</em> developers being judged in their absence</strong>, on the quality of their git logs. For example when deciding whether to let someone go who is considered sub-standard for the team, the principal engineer and an engineering manager might go over the individual's output in private, without that individual present, to collect some evidence and inform their final decision. Do you want that to be you? Honestly, that "wip" commit with your name & email on it that seemed so harmless at the time doesn't look so good now it's out of context of whatever rush / justification there seemed to be in the moment. Worse still if you make a regular habit of lazy / rushed / incoherent commits.</li>
</ol>

"Write every commit message like the next person who reads it is an axe-wielding maniac who knows where you live" ~ unknown</p>
</blockquote>
Why the business owner cares about good commit messages</h2>
(Even if they don't know it)</p>
Developers writing high-quality commit and pull request (PR) messages provides clear context and explains the "why" behind changes, directly impacting the clarity and long-term value of the codebase as an asset.</p>
Well-crafted messages ensure that future developers, including new hires, can quickly understand the rationale behind decisions, saving time, reducing the cost of onboarding and reducing the risk of misinterpretations that could lead to costly errors.</p>
They support smoother collaboration across teams, foster accountability, and create a historical record that aids in audits, troubleshooting, and strategic decision-making.</p>
This practice minimizes the risk of long-term inefficiencies, speeds up development, and ensures technical work aligns with broader business objectives.</p>
Why I care about good commit messages</h2>
For reasons of chance I've ended up more than once maintaining and extending things created by other people, many of whom had moved on from the projects and were no longer contactable.</p>
When dealing with a piece of code that has a behaviour that is clearly causing problems for users/customers etc. sometimes I have needed to understand why</em> it was that way before I could know whether it could be changed without creating even bigger problems.</p>
Another example was a buggy and complex algorithm implementation, but no clue left as to what the algorithm being implemented was, as a result a week of reverse-engineering the maths happened that could have been saved with a simple "this is an implementation of ..." with a hyperlink or algorithm name.</p>
With the author of the code not available to ask I'm left only with the source code and git blame</code> to fathom why it's like that.</p>
When you find the commit that added the patch and it's one of the following then you realise the author was not taking into account future maintainers:</p>

"wip"</li>
an entire fully formed set of behaviour appears in one giant commit with no explanation (perhaps copied from somewhere else)</li>
a mention of a ticket number from a defunct ticket tracker with no further explanation</li>
</ul>
It seems to me that it doesn't take much additional effort to rattle out a sentence or two with some context on why something is being changed for the benefit of future maintainers. Especially when it's a complex patch that maybe took more than a day to create.</p>
Given that each line of code is read many more times than it is written it seems that being "lazy" with explanation is not delivering the highest quality output to your client/employer/project.</p>
I, like many developers, also value speed of delivery, fast iteration, early prototypes that may get rewritten; but you can still move fast while taking a few minutes for each patch to explain it. And you have to think that just because you might throw this one away, you also might not, and you just don't know yet.</p>
Why good commit / pull-requests matter in commercial delivery</h2>
I often run into the objection of "we don't have time for polishing git logs" in the context of commercial projects where time is money.</p>
In commercial delivery teams good commits and pull requests are every bit as important. High quality commits & pull requests result in faster and more effective peer review, with less risk of missing important problems.</p>

Clear intent (i.e. "why this change now") reduces review time. - Reviewers don’t have to reverse-engineer the purpose of the change, they can validate correctness quickly instead of spending time guessing.</li>
Small, single-purpose commits isolate risk. - When refactors and behaviour changes are separated, reviewers can approve mechanical changes quickly and focus attention only where logic actually changed.</li>
Descriptive messages enable reliable approval / sign-off. - A reviewer can confidently understand and confirm that "this change does X to solve Y".</li>
Readable history accelerates future fixes. - When something later breaks or needs extension, engineers can locate the exact change and rationale quickly, shortening incident resolution and feature lead time.</li>
</ul>
Well-structured commits are a throughput optimization: they reduce review latency, improve the chances of catching problems early and shorten future debugging time - all of which speeds up the delivery of value rather than slowing it.</p>
Pull request descriptions matter too</h2>
It is common to ensure peer review by way of "pull requests". Pull request descriptions are a lot like commit messages - they are used to describe to another human the contents of change to the codebase.</p>
As such the pull request descriptions perform largely the same function as commit messages, and should follow the same guidelines laid out here for the same reasons.</p>
In the hierarchy of longevity and accessibility they sit between ticketing systems and git commits. Git logs will likely outlive both pull requests and tickets.</p>
If you make an effort to make the git commits contain sufficient context for the reader of a patch then it's often useful to copy-paste that information into the pull request description, tidy it up for the reader and a bit more context such as screenshots.</p>
How to make them better</h2>
There's not much for me to add on what's already been written, so read these articles on the specifics of writing good commit messages.</p>

Use the present-tense imperative</a> ("Add …" not "Adds" or "Added")</li>
https://vip.wordpress.com/documentation/commit-messages/</a></li>
https://robots.thoughtbot.com/5-useful-tips-for-a-better-commit-message</a></li>
http://chris.beams.io/posts/git-commit/</a></li>
https://gds-way.cloudapps.digital/standards/git.html</a></li>
https://mislav.net/2014/02/hidden-documentation/</a></li>
Pull Request Etiquette gist by mikepea</a> - covers pull request quality as well as individual commits</li>
</ul>
My personal additions to this list:</p>

A list of highlights of changes in bullets is often nice to add, think of it as a tourist's guide to your patch. It makes it easier to spot the key changes in a large diff, and can make code-reviews more effective.</li>
Hard-wrapping lines shouldn't be required, that should be an editor/display concern but unfortunately the git tooling doesn't agree so doesn't wrap anything so you might have to hard-wrap.</li>
</ul>
Are you expecting every tiny change to be like this?</h2>
No. Some patches really don't need much explaining, e.g. re-applying default code formatting or fixing a typo; but you should always consider what context a future reader might need.</p>
Are your patches atomic incremental improvements</h2>
If it's hard to write a good message, it might be that you are not taking the time to craft good single-purpose commits.</p>
Examples of good</h2>

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f076ef44a44d02ed91543f820c14c2c7dff53716</a> (via reddit</a>) - what I like about this one is:

It adds context</strong> that you could never get from code (note some is repeatedly more briefly in code comments which is a good thing)</li>
It explains the new behaviour in human terms</strong>.</li>
It's easy to read (good quality English prose)</li>
</ul>
</li>
https://github.com/DFE-Digital/find-teacher-training/pull/159/commits/00e24dbc216836dd73281688491b8da355706d81</a> - what I like about this one is:

It's part of a PR that is also well described & reviewed</li>
It adds context (about the thing that will call the endpoint added in the patch, i.e .the reason it was created)</li>
It mentions a PR in another repo that was a source for some of the code & ideas, yet more context for answering the question "why was this done and why is it like this?"</li>
The co-author is attributed (github shows this which is nice), this might give you someone to talk to about if they're still around</li>
It provides an outline of the patch so you don't have to parse the whole diff to get a flavour of how the patch changes behaviour. When you have a lot of patches to read because you're looking for something in the history this can be a big timesaver.</li>
It gives insight into why certain decisions were made about the final shape of the patch (e.g. why just /healthcheck and not /ping as well)</li>
</ul>
</li>
</ul>
Video presentations on why good history matters</h2>
Remember, your code and your commits can last a veeeeeery long time and you never know what poor soul will have to understand what you did and why years later... when you have time watch this video:</p>
</iframe>
</iframe>
Adding ADRs</h2>
Documenting broader reasons can be done with ADR</a>s (Architecture Decision Records, i.e. why we did this and not that), Y-ADRs</a> (basically highly condensed one-liner ADRs), readme files etc., ideally in ascii</a>, and even more ideally in the same commit so that a git blame</code> will lead the future reader to find the additional reasoning right there in the same patch as the change to the code.</p>
Learning more about git</h2>

NDC Talk: "git beyond pull & push" - Jørgen Kvalsvik - NDC TechTown 2023</a></li>
</ul>


Starting up a startup
2015-12-24T18:06:00.002+00:00
I've been contracting for 3 years now, which fits nicely with my drive to have real customers and build business(es). But I still have an itch. I've now found something to scratch that itch (taking the metaphor too far, sorry about that).</p>
I'm going to write about my latest adventure here, from the perspective of me as a coder who's aspiring to be an entrepreneur, so if you're interested in my personal journey on this then do add my blog to your rss reader etc. of choice. I'll try and tag them all with startup so you can filter if you aren't interested in my other ramblings :-)</p>
This post tells the story how I got involved in this startup, and some of the technical choices made so far and what we have in place already.</p>
So here's how it started for me:</em></p>
I found out via twitter that the the startup britain bus</a> was coming to Reading, so I went to say hi. They were a friendly crowd, and whilst there I found out that there's actually a new startup group in the area called ConnectTVT</a> associated with the new co-working space called grow@greenpark</a> over in green park just off M4 J11. That beats traipsing into the big smoke for me and is just the kind of melting pot I've been looking for. I know plenty of tech folks given that I'm a professional coder, but it doesn't seem sensible for me to start a business with people with the same skills and the same blindspots as me. None of my "normal" friends seem to have the same entrepreneurial itch that I do; I needed a broader pool. I went straight from the bus to grow to check it out and met some more lovely friendly people. A few weeks later (I think) I went back again for one of their startup grinds to see what was going on. There was a small group of interesting people with diverse interests and talents, from accountants to designers and a few tech folk. One of the people there, Richard</a>, had an elevator pitch for a new take on the property market which made some sense to me and seemed well received by the group. All the people there offered their various advice and help. Richard has had experience in the property market that's led him to this insight and having mulled over the idea for quite some time was interested in how this concept might become something more real. Lacking the tech background to just go and build it himself he was interested in what kind of scale of task this might be and how to find people who could help. I offered to provide advice any time and passed Richard my contact details. I didn't at that point have any plans to get further involved.</p>
I very much enjoyed meeting everyone, and went home with a head full of ideas and inspiration. I found myself still thinking it all through in the small hours that night, regularly grabbing my phone to make yet another note on the trello</a> app detailing some idea about this property thing and how it could work, how you might get over the network-effect bump etc. Clearly this wasn't going away. So I messaged Louize</a> who runs the meetup to get in contact with Richard and said it had been keeping me awake with ideas, and she got us in contact again.</p>
After mulling the whole thing over for a bit longer, it occurred to me that there is no reason I couldn't do more than just provide advice, and maybe I should offer to provide a more concrete involvement. After all, I think the idea is great, and it's not like I'm busy making any of my other ideas I have into businesses yet. Richard would certainly be much better able to build at a much lower financial risk with a technical co-founder. So after a load more discussions around the idea, the odd trip to the pub and some discussions around practical arrangements that is now what we are doing.</p>
How's it going?</em></p>
The first thing has been to really understand Richard's vision, this has mostly involved talking lots. I set up a trello board for the startup, and that's been great for doing a brain dump of all the ideas and todos, and getting them into some sort of organisation, then moving them into "do later" or "needed for launch" kind of groupings. I've also pointed Richard at balsamiq mockups</a>, which these days has a slick web based editor and presentation mode. I've used it in the past to great effect when helping stakeholders in other projects understand what they're really likely to get before all the expense of building the wrong thing. In this startup it's already been really useful for hashing out ideas of how the site might work and even for getting reactions and insight by way of putting it in front of people and seeing if they understand the concept and how you would use it if it were a real thing already. Inspiration for this process is taken from the excellent book Don't Make Me Think</a>.</p>
Hosting choices...</em></p>
In terms of building something real, Richard bought a domain, so we have somewhere to put it. I spent forever over technical choices</a> because basically you could</em> use just about any technology stack, any programming language, cloud PaaS, IaaS or private servers etc. and in fact they would all work. There's actually nothing to completely rule out anything. Some great things have been built on top of apparently terrible technical choices (wordpress is lovely but php that it's built with is known for being buggy and hacky). There's much stock put in "cloud" these days, and it does have benefits such as infinite scalability (of your wallet), but it comes with an overhead of complexity and potentially vendor lockin (though that like everything can be mitigated). So call me a luddite but I've gone for an old fashioned linux VPS (virtual private server) where I can easily put a ceiling on costs till we have an income. Unlike so many modern startups that rely on farming people with free shiny stuff until they have enough they can magically make money off the back of them, burning VC cash all the while in the name of profitless growth, we're going to provide users with value, and charge them appropriately for that value, which naturally puts the brakes on the kind of explosive growth of free users that can overwhelm an ambitious startup's site. So I've made a decision at last, hunted around for a provider I could trust and bought some hosting. There's a lot to choose from so I needed a way to narrow down the field, especially given there's a lot of not very good and not very well supported VPS offerings out there. I thought there might be an advert in Linux Voice</a> magazine, which is where I found my uber-geeky ISP A&A</a> (check out the boss's blog</a> if you want to know how far that rabbit hole goes), but there weren't any. So I hopped on #linuxvoice on freenode irc and asked in there, and sure enough the good folks at the mag said they'd been using bytemark</a> and had been happy with their support. Sold.</p>
So I bought bytemark's basic VPS offering, and now have a server on the internet that I can point everything at and run everything we need on no matter how quirky the needs. While we have a manageable number of users this should suffice, we're not planning any multi-million pound TV adverts any time soon. I expect if we're successful we'll have growing pains no matter what platform we're using, it's non-trivial to scale even if you're on a so-called scalable platform in the first place. There's lots of gotchas, unexpected interactions and bottlenecks in any system as it grows.</p>
If it sounds like I'm trying to rationalise my decision that's because I am, I'm aware that whatever technical (and even non-technical) decisions are made at this earliest of early stages have rippling effects that are hard to reverse once you build more on top of them. However equally no matter which choices you make, none of them are truly irreversible, and there are myriad ways of tackling any of the problems that come your way in time. I've had the privilege of seeing teams that are running decent scale operations in both .net and ruby on rails, which are two very different platforms with different cultures and different sets of problems, but interestingly the difference in the overall scale of the challenge and how easy it is to get things done in the long haul was not noticeably different as far as I could tell.</p>
Mailing lists and landing pages:</em></p>
The first goal is to have a way to let interested people know when we've got things to show them, tell them the story, and when we need their help. That means a mailing list. Bulk email isn't for the feint hearted these days in the modern world of aggressive spam filters, SPF (no not sunblock), domain keys and increasingly rigid legal constraints as we try to fight outright spam on the wild west of the internet so I don't fancy doing it all myself. My go-to service for this is mailchimp</a>, and I haven't been disappointed. But for this to work we need an email address, and that means MX records and pointing it at some mailserver somewhere. I did look into gmail as a possible shared mailbox but it doesn't seem to do what we're after, so now I have a server running anyway I can just use that. No artificial limits, no snooping, no unwanted advertising.</p>
It turns out that bytemark provide a very nice customised build of debian called symbiosis</a>, which has a bunch of useful services already installed, and a watchdog service that'll restart anything that's fallen over. Nice! It didn't take me much fiddling to get the basic web hosting to behave and get an email account on it set up. I then used the domain control panel to point the A & MX records at our shiny new server (by way of a bit of chaining of CNAMES which will be slightly slower for the first lookup but will be easier to maintain). I then had to redo the web and mail hosting because I hadn't realised that bytemark had made the server do multiple domain hosting using a neat folder based system, but that didn't take too long and it's all good learning time without customers breathing down one's neck!</p>
I've now done a rudimentary holding page for the domain, with a sign-up form for the mailing list, which was goal number one. You can see it here: http://www.xchain.co.uk/</a>. If you're interested in the concept or progress then please do sign up to the mailing list. It's easy to unsubscribe if you get bored of it ;-)</p>
I've added google analytics</a> to the site, which is very useful albeit slightly evil, so at least we'll have some idea if it's getting any traction as we start to spread the word (taking all stats on the internet as extremely approximate of course). I might switch to one of the more libre solutions at some point but that would require a bit more time and this is after-all a commercial venture and not an open source project. Much as I like working on open source for the greater good, I haven't yet managed to make it pay the bills. I think it's important to have metrics for a startup so you can tell if what you're doing to spread the word is actually having any effect, and what might be putting off potential customers.</p>
We've sent out a mailing or two to the few people we've got signed up at this early stage (hello mum!) as much for practice as anything, and I'm impressed with mailchimp's collaborative capabilities and previewing system.</p>
Tech stack</em></p>
You may have noticed I still haven't entirely picked a stack, nothing I have so far has locked me in to a decision. I don't want to be paying for or locked in to Microsoft's world (and the associated treadmill of change), so although I do their stuff for a living I don't want to tie my own startup to them, so that's out. I'd prefer an open source stack as I think it matters whether you enjoy working on the platform you've chosen, and open source is just so much nicer to work with as a coder / dev-ops. Currently I'm thinking a golang based server, backed with postgresql (the industry's finest database) serving up a JSON/REST API to an AngularJS + Foundation frontend. This gives us scope to rework the different parts as needed, and to bolt on dedicated mobile apps later.</p>
Web design</em></p>
There's no design in the current holding page as I am no designer (frustratingly, but that's a whole other career path), and we're still working out how to pull design talent into the whole shebang in a sustainable way. Richard has put me onto webflow</a>, which maybe will get us started.</p>
My first learning of trying to find a freelance designer was that I need a "web designer" not just a "designer", being from the web world I hadn't appreciated the importance of the prefix and attracted all the wrong people.</p>
Further reading...</em></p>
Having this startup has encouraged me to broaden my reading and listening habits further. Here's some of things I've been catching up with:</p>

Regret minimization vid</a> - inspiration from Bezos</li>
What is code</a> - explains what I do to the non-coders</li>
HanselMinutes - Startup Accelerators</a> - a timely podcast show</li>
The Startup Chat podcast</a> from WPCurve</li>
Last year's startup resolutions</a> via the work in startups</a> mailing list</li>
Everything in LeanStack's blog</a></li>
</ul>
--</p>
Please do drop me a line if you found this interesting in any way. tim@timwise.co.uk</a> works best for me.</p>


Sql Data Viewer - preview release
2015-10-27T14:45:00+00:00
Hello dear readers,</p>
I've started working on a data brower for relational databases that allows you to click through ids with foreign-keys to get around.</p>
Take a look at http://www.timwise.co.uk/sdv/</a> and if you think you might be interested then please do sign up to my mailing list</a>, and let me know if you have any feedback.</p>
This started as a holiday exercise in learning google's go programming language, but is intended to become a fully-fledged software product available to buy.</p>
Tim</p>


100 reasons I hate ssrs
2015-08-12T19:01:00.004+00:00
Originally posted at https://timwise.blogspot.com/2015/08/100-reasons-i-hate-ssrs.html</a> which attracted some great comments.</em></p>
A slightly tongue-in cheek hit-list of nasty things and vague hand-waving
opinions on what makes microsoft's sql server reporting services (ssrs) such a
pig to work with.</p>
I don't really know of anything better so this is mostly just pointless
ranting; but I'll justify it to myself by saying at least you'll know what
you're getting into if you've read this before you start. SSRS seems to be more
"death by one thousand paper cuts" than completely broken, so it's not so easy
to say "it's shit, shalln't use it" like any good prima-donna developer would.
Sorry I mean rock-star (recruiter speak). It also offers a few features that
would be pretty hard to code from hand cost-effectively in something like
asp.net mvc, such as user editing, multiple export formats, scheduled emails,
and some of the ways you can cut-and-shut the data in the reports.</p>
I make no apologies for the colourful language, it's a representation of the
emotional side of having to use this heap of crap.</p>
Some of these were contributed by in the comments (on blogger) after I posted
the original article. (Thanks, that was a nice surprise, and it's nice to know
it's not just me!)</p>

http://www.allenkinsel.com/archive/2013/01/adventures-in-ssrs</a></li>
doesn't bind to a port like a normal fukcing service

http://blogs.devhorizon.com/reza/2008/10/20/say-goodbye-to-iis-say-hello-to-httpsys/</a></li>
http://www.iis.net/learn/get-started/introduction-to-iis/introduction-to-iis-architecture#Hypertext</a></li>
https://social.technet.microsoft.com/Forums/sqlserver/en-US/f2586aca-78fe-40d6-9bcd-5151bac7136f/role-of-httpsys-in-ssrs-2008-?forum=sqlreportingservices</a></li>
http://blogs.technet.com/b/andrew/archive/2007/12/04/sql-server-2008-reporting-services-no-longer-depends-on-iis.aspx</a></li>
http server api (aka http.sys)

https://msdn.microsoft.com/en-us/library/aa364510%28VS.85%29.aspx?f=255&MSPPError=-2147217396</a></li>
</ol>
</li>
list reservations:

netsh http show urlacl</code></li>
</ol>
</li>
</ol>
</li>
auth in reporting https://msdn.microsoft.com/en-us/library/ms152899.aspx</a></li>
reports in VS

http://curah.microsoft.com/22200/create-ssrs-reports-using-visual-studio</a></li>
https://msdn.microsoft.com/en-us/library/ms173745.aspx</a></li>
</ol>
</li>
"Explicity add new role assingment for the account you are using and check every box in sight" ~ a.n. colleague.  lol</li>
ignore the .rdl.data files with git.

http://stackoverflow.com/questions/3424928/in-ssrs-is-there-a-way-to-disable-the-rdl-data-file-creation#3425429</a></li>
</ol>
</li>
no folders.

https://connect.microsoft.com/SQLServer/feedback/details/487106/allow-sub-folders-in-ssrs-projects</a></li>
http://stackoverflow.com/questions/3309002/visual-studio-for-ssrs-2008-how-to-organize-reports-into-subfolders-in-solutio</a></li>
</ol>
</li>
renamed a dataset, nothing fucking broke!!!!!!!!!!!!!!!!!!!!!!!!!!! even though there are reports that depend on it. On editing the report's dataset list you can see clearly "not found", but yet it still runs. what in the blazes is that all about?

caching in the report editor http://stackoverflow.com/q/3424928/10245</a></li>
kill the .data cache files http://stackoverflow.com/questions/3424928/in-ssrs-is-there-a-way-to-disable-the-rdl-data-file-creation</a></li>
git clean -xfd</li>
fuck</li>
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/0aa81692-352f-4c1f-a0e3-95fe6c0797ca/cachedataforpreview-in-rsreportdesignerconfig-not-honored</a></li>
https://connect.microsoft.com/SQLServer/feedback/details/468482</a></li>
it's the bin\</code> folder, not the .data files. Still, fuckkkk.</li>
</ol>
</li>
to get from a report to a db you go, report > report dataset > shared dataset > db, but db is defined in the shared dataset with another name, which can be pointed to a shared data source, which is also named. and that</em> data source actually has a connection string</li>
committing to tfs failed half way through because vs had locked a bunch of files I didn't even have open</li>
found a param with <Value>=Microsoft.VisualBasic.Strings.Join(Parameters!Stages.Label, ", ")</Value></code> - wtf.</li>
function overload matching warning wouldn't go away till I closed the sln</li>
localisation is a bitch

http://stackoverflow.com/questions/10953629/how-to-change-ssrs-2008-locale</a> etc</li>
http://blog.ponytailbob.com/2007/10/multi-language-tips-in-ssrs.html</a></li>
http://blogs.msdn.com/b/sriram_reddy1/archive/2012/01/09/localization-in-ssrs-reports.aspx</a></li>
https://support.microsoft.com/en-gb/kb/919153</a></li>
http://www.keepitsimpleandfast.com/2011/09/localization-of-your-ssrs-reports.html</a></li>
why you no use User!Language??</li>
</ol>
</li>
Visual Studio 2013 crashed. hard. while cancelling new report param</li>
adds 00:00:00 to date fields from sql server. duuuuuuuuuuuuh (goes via .net datetime internally, but even so, not friendly)</li>
changed date format, looks fine in VS, but no change in report server. wuh? deploy all</li>
no auto-sizing of cols http://stackoverflow.com/questions/7851045/ssrs-tablix-column-cangrow-property-for-width</a></li>
no nulls in multi-value http://www.keepitsimpleandfast.com/2012/03/how-to-pass-null-value-to-multi-value.html</a></li>
need dirty hack to show "all" rather than full list

http://www.mssqltips.com/sqlservertip/2844/working-with-multiselect-parameters-for-ssrs-reports/</a></li>
http://stackoverflow.com/questions/12917261/optional-multi-valued-parameters-in-ssrs</a></li>
http://www.bi-rootdata.com/2012/09/efficient-way-of-using-all-as-parameter.html</a></li>
http://blog.ponytailbob.com/2007/10/2-shortcomings-of-multi-valued.html</a></li>
</ol>
</li>
some fucking horror I've yet to encounter (querystrings) http://stackoverflow.com/questions/512105/passing-multiple-values-for-a-single-parameter-in-reporting-services</a></li>
it has a fucking canvas size that will push over to 2 pages</li>
the font kerning on a print is massively different to on web / design view</li>
sorting

http://stackoverflow.com/questions/9254604/why-does-my-sql-server-reporting-service-ssrs-report-appear-to-re-sort-the-d</a></li>
</ol>
</li>
"Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index " from editing xml. thanks for the error info. fuckkkkkers</li>
the ssrs gui editor is a flaky piece of shit

doesn't select the right fucking textbox in the props window</li>
had to restart visual fuckigjn studio</li>
grrr</li>
</ol>
</li>
the underlying xml is fucking horrific</li>
<rd:Selected>true</rd:Selected></code> ----- what in the fucking blazes is that doing in there?</li>
how do you deploy without connecting visual studio to production server? you fucking don't hahahahahaa</li>
powerhell https://gist.github.com/timabell/7e3019bd2de802f0b259</a></li>
ssbi install croaked - https://support.microsoft.com/en-us/kb/2800050?wa=wsignin1.0</a></li>
[09:44:53] john doe: Tim can I have a .bak</code> file of ReportServer$MSSQL2012TempDB</code> which the stupid software seems to be unable to operate without even though it has Temp</strong> in the database name implying it will rebuild itself (at least that's what it implies to me)

[09:47:48] Tim Abell: (facepalm)</li>
[09:47:49] Tim Abell: sure</li>
[09:48:08] Tim Abell: I did wonder, and then I thought, no they couldn't possibly need that</li>
</ol>
</li>
dropdown doesn't work in firefox</li>
no debugging http://stackoverflow.com/a/14068447/10245</a></li>
Warning : The text box ‘appliedFilters’ and the image ‘urLogo’ overlap. Overlapping report items are not supported in all renderers.</li>
the only options for DRY in reports suck balls http://www.3pillarglobal.com/insights/tips-tricks-ensure-consistency-sql-server-reporting-services-reports</a></li>
http://harmful.cat-v.org/software/xml/</a> xml is a terrible format anyway</li>
layout is in inches</li>
you can change the layout to cm</li>
it stores different metrics (cm/in) for each element, wtf, pick a unit</li>
reflowing nicely is impossible</li>
layout engine is as intelligent as a piece of paper and a pen. x,y is all you get.</li>
http://www.webapplicationsuk.com/2010/07/word-html-renderer-ndash-the-road-to-hellhellip/</a></li>
https://connect.microsoft.com/SQLServer/feedback/details/540183/supported-rdl-object-model-rdlom</a></li>
this is the kind of bullshit that counts for helpful content on the net https://social.msdn.microsoft.com/Forums/en-US/86205ca4-13d0-4ca6-84f1-79797616f0f4/exclude-null-values-from-sum-and-avg-calculation?forum=sqlreportingservices</a> - =sum(forum_format * quality_of_community)</code> = errorrrrrrrrr</li>
multiple rdl xml schema in the same fucking project, completely different xml structure

2005 generated with "new report wizard" in VS 20-fucking-13: <Report xmlns="http://schemas.microsoft.com/sqlserver/reporting/2005/01/reportdefinition" xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"></code></li>
2008 <Report xmlns="http://schemas.microsoft.com/sqlserver/reporting/2008/01/reportdefinition" xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"></code></li>
2009 from https://technet.microsoft.com/en-us/library/cc627465%28v=sql.105%29.aspx</a> - <Report xmlns:rd=http://schemas.microsoft.com/SQLServer/reporting/reportdesigner xmlns="http://schemas.microsoft.com/sqlserver/reporting/2009/01/reportdefinition"></code></li>
2010 <Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns:cl="http://schemas.microsoft.com/sqlserver/reporting/2010/01/componentdefinition" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/01/reportdefinition"></code></li>
how many fucking versions??!</li>
http://stackoverflow.com/questions/15539859/what-is-the-difference-between-rdl-2008-schema-and-rdl-2010-schema-feature-wise</a></li>
http://stackoverflow.com/questions/9974179/is-there-a-new-version-of-rdl-schema-for-sql-server-2012-denali</a></li>
http://stackoverflow.com/questions/29951653/ssrs-2008r2-visual-studio-2008-and-2008-and-2010-schemas</a> - how to not end up with old schema?!</li>
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/f4d14548-c592-4d8d-8185-ca683c421649/2010-schema-with-visual-studio-2010?forum=sqlreportingservices</a></li>
</ol>
</li>
how do you upgrade a report schema? install a massive chunk of fucking sql server http://stackoverflow.com/questions/13170608/upgrade-my-rdlc-schema-from-2008-01-to-2010-01</a></li>
[17:54:48] john doe: Btw have you noticed that in Print Layout view the header doesn't expand if any of the textboxes have auto-grown? [17:55:14] Tim Abell: that's because ssrs is a piece of shit from 1990 [17:55:28] Tim Abell: and it thinks A4 is the ultimate display format [17:55:47] Tim Abell: you just have to guess how much space you'll need</li>
the ordering of the xml in the proj file is unstable causing diff noise</li>
subreports, icky http://bhushan.extreme-advice.com/subreport-in-ssrs/</a>

or nested tables http://stackoverflow.com/questions/11335655/filtering-nested-data-regions-in-ssrs</a></li>
</ol>
</li>
some things can only be achieved with subreports, and they have to be deployed separately from the main report, meaning they can get out of sync. enjoy the fear of not knowing if you'll break something else when you upload your new version of the subreport you depend on</li>
no support for "time" data type http://stackoverflow.com/questions/3846378/displaying-time-in-reporting-services-2008</a></li>
The "View Report" button next to the parameters when running a report in VS is always</em> greyed-out, even though it actually works.</li>
wow that's mental, hidden reports show in details view and not in tile view in the ssrs web ui</li>
the report editor has a copy option for report items, but no paste, so you can't duplicate reports

actually you can, but only if you know the keyboard shortcuts. 0_o - ctrl-c ctrl-v</li>
</ol>
</li>
duplicating a report on the filesystem, and then using "add existing item" to include it puts it at the end of the list... until you rename it and then it's moved into alphabetical order causing a spurious diff. should have put in the right place in the first place. grr.</li>
using the cursor keys to move textboxes around is so laggy that I overshoot every single time</li>
the editor popups in visual studio are modal, so you can't refer to anything else</li>
and there's no maximise button so you have to drag the fiddly border to make it bigger</li>
the report editor hasn't heard of ctrl-c or ctrl-v, have to use ctrl-Ins / shift-Ins instead</li>
border rendering / precedence is a fucking mess. set some borders, your report will look like a two-year-old coloured it in, and how it looks changes depending on the zoom level.</li>
you have to use VB to do alternate row colours - http://stackoverflow.com/questions/44376/add-alternating-row-color-to-sql-server-reporting-services-report</a></li>
if anything goes wrong with an expression all you get is "#Error". Helpful. E.g. http://stackoverflow.com/q/9144312/10245</a></li>
this one time, at band camp, I edited a report definition in VS and it refused to save the new definition to disk (ctrl-shift-s, ctrl-shift-s!!). wtf. Restarted VS and all the changes were gone.</li>
report editor silently adds new parameters to the report when you add new parameters to the sql. seriously. fuck off.</li>
RSI-inducing UI for editing the reports. click click click clickity click</li>
the sql editor has only a single undo. like ye olde notepad.</li>
in the editor, you can right-click copy, you can't right-click paste. wtf. ctrl-v does paste though. wtf again. I know, I already said it, but it's reaaaaaly shit</li>
the field list on a dataset is ordinal, allowing you to mismatch the select in the sql from the list of fields in the dataset and not notice</li>
how do you align a textbox on the page?<</span>Top</span>></span>0.82546cm</span></</span>Top</span>></span></span>
<</span>Left</span>></span>0.07309cm</span></</span>Left</span>></span></span>
<</span>Height</span>></span>0.88964cm</span></</span>Height</span>></span></span>
<</span>Width</span>></span>2.3025cm</span></</span>Width</span>></span></span></code></pre>fuck you!!!!</li>
one goddam cell in the underlying format:<</span>TablixCell</span>></span></span>
  <</span>CellContents</span>></span></span>
    <</span>Textbox</span> Name</span>=</span>"</span>qty</span>"</span>></span></span>
      <</span>CanGrow</span>></span>true</span></</span>CanGrow</span>></span></span>
      <</span>KeepTogether</span>></span>true</span></</span>KeepTogether</span>></span></span>
      <</span>Paragraphs</span>></span></span>
        <</span>Paragraph</span>></span></span>
          <</span>TextRuns</span>></span></span>
            <</span>TextRun</span>></span></span>
              <</span>Value</span>></span>=Sum(Fields!qty.Value)</span></</span>Value</span>></span></span>
              <</span>Style</span> /></span></span>
            </</span>TextRun</span>></span></span>
          </</span>TextRuns</span>></span></span>
          <</span>Style</span> /></span></span>
        </</span>Paragraph</span>></span></span>
      </</span>Paragraphs</span>></span></span>
      <</span>rd</span>:</span>DefaultName</span>></span>qty</span></</span>rd</span>:</span>DefaultName</span>></span></span>
      <</span>Style</span>></span></span>
        <</span>Border</span>></span></span>
          <</span>Color</span>></span>LightGrey</span></</span>Color</span>></span></span>
          <</span>Style</span>></span>None</span></</span>Style</span>></span></span>
        </</span>Border</span>></span></span>
        <</span>PaddingLeft</span>></span>2pt</span></</span>PaddingLeft</span>></span></span>
        <</span>PaddingRight</span>></span>2pt</span></</span>PaddingRight</span>></span></span>
        <</span>PaddingTop</span>></span>2pt</span></</span>PaddingTop</span>></span></span>
        <</span>PaddingBottom</span>></span>2pt</span></</span>PaddingBottom</span>></span></span>
      </</span>Style</span>></span></span>
    </</span>Textbox</span>></span></span>
  </</span>CellContents</span>></span></span>
</</span>TablixCell</span>></span></span></code></pre></li>
Report width limited to 455in (even if I'm hiding columns using parameters against the Visibility column filter).</li>
NO DYNAMIC COLUMN CREATION (ridiculous!)</li>
Selection of multiple columns and setting attributes is ridiculously flaky. This is because I wanted to reduce the column width and font to comply with Point 71 (max report width)!!</li>
We have to restart Reporting Server services frequently or our charts won't show up. Eh?</li>
Cut and paste columns? Nope!</li>
Disappearing "Report Data" menu: http://stackoverflow.com/a/28883272/1024</a></li>
SSDT for VS2015 upgrades reports to 2016 schema as soon as they're opened (WAT?!)  and ignores the TargetServerVersion being set to <=2014 http://stackoverflow.com/questions/37816216/deploy-of-a-report-with-ssdt-2016-generates-error</a> what if we haven't upgraded our production server, hmmm?</li>
Okay so when you run a build, VS2015 then downgrades</strong> the rdl to the right schema to match the project's target server version before putting it in the build folder. What could possssssibly go wrong. 0_o   Complexity++</li>
Install SSRS in only 41 easy steps https://thecodeattic.wordpress.com/category/ssrs/</a></li>
Build > "Skipping 'SomeReport.rdl'. Item is up to date." - No it isn't, I've deleted it from the friggin bin folder. So clearly it has some stupid cache of what's it thinks is on disk rather than, oh I don't know fucking checking the disk. Sheesh. If it's that optimized why is everything still so damn slow?!</li>
Building a project with lots of reports is slow. Even if nothing changed.</li>
Intermittently get "[rsInvalidReportDefinition] The definition of this report is not valid or supported by this version of Reporting Services. The report definition may have been created with a later version of Reporting Services, or contain content that is not well-formed or not valid based on Reporting Services schemas. Details: Data at the root level is invalid."</em> - I am not alone. https://connect.microsoft.com/SQLServer/feedback/details/2988044/randomly-get-rsinvalidreportdefinition-when-previewing-report</a> - A rebuild fixes it for me, for a while at least.</li>
Sometimes when you edit an embedded dataset it completely fails to persist any of your changes to disk. And when you close the report they are lost. Handy. Thanks fuck for git.</li>
Assumes you've never heard of source control and creates numbered backups of report files (notably on auto-upgrade). Also has messages like "delete will permanently delete this thing</em>" - no it won't I have source control; I wonder if the SSRS know what that is.</li>
I've never managed to crash VS2015 so many times in one day. Omg you didn't click there did you? I wasn't ready! crash</em> ... again</em></li>
Change the files on disk and the preview window often doesn't notice</li>
Preview window silently fires a 'build' of the reports. Sometimes.</li>
Generates broken Shared-datasets http://stackoverflow.com/a/38753141/10245</a></li>
If you break the xml of a shared-dataset the entire project</em> will fail to load. w-t-f.</li>
Renaming datasets etc just breaks everything rather than updating references.</li>
The .rptproj</code> file has a <state></code> tag at the top which is base64 encoded xml (WAT?! xml in xml. eerrrrr), which is information about source control (arse-about-face or what! source code that controls the source-control</em> [that controls the source, that controls the source-control that controls the source...]; even the words are circular!!</em>). In the base64 you'll find a <SourceControlInfo></code> tag.<</span>Project</span> xmlns</span>:</span>xsi</span>=</span>"</span>http://www.w3.org/2001/XMLSchema-instance</span>"</span> xmlns</span>:</span>xsd</span>=</span>"</span>http://www.w3.org/2001/XMLSchema</span>"</span> ToolsVersion</span>=</span>"</span>2.0</span>"</span>></span></span>
<</span>State</span>></span>$base64$PFNvdXJjZUNvbnRyb2xJbmZvIHhtbG5zOnhzZD0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEiIHhtbG5zOnhzaT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEtaW5zdGFuY2UiIHhtbG5zOmRkbDI9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDAzL2VuZ2luZS8yIiB4bWxuczpkZGwyXzI9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDAzL2VuZ2luZS8yLzIiIHhtbG5zOmRkbDEwMF8xMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDA4L2VuZ2luZS8xMDAvMTAwIiB4bWxuczpkZGwyMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEwL2VuZ2luZS8yMDAiIHhtbG5zOmRkbDIwMF8yMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEwL2VuZ2luZS8yMDAvMjAwIiB4bWxuczpkZGwzMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDExL2VuZ2luZS8zMDAiIHhtbG5zOmRkbDMwMF8zMDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDExL2VuZ2luZS8zMDAvMzAwIiB4bWxuczpkZGw0MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEyL2VuZ2luZS80MDAiIHhtbG5zOmRkbDQwMF80MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEyL2VuZ2luZS80MDAvNDAwIiB4bWxuczpkZGw1MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEzL2VuZ2luZS81MDAiIHhtbG5zOmRkbDUwMF81MDA9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vYW5hbHlzaXNzZXJ2aWNlcy8yMDEzL2VuZ2luZS81MDAvNTAwIiB4bWxuczpkd2Q9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vRGF0YVdhcmVob3VzZS9EZXNpZ25lci8xLjAiPg0KICA8RW5hYmxlZD50cnVlPC9FbmFibGVkPg0KICA8UHJvamVjdE5hbWU+U0FLPC9Qcm9qZWN0TmFtZT4NCiAgPEF1eFBhdGg+U0FLPC9BdXhQYXRoPg0KICA8TG9jYWxQYXRoPlNBSzwvTG9jYWxQYXRoPg0KICA8UHJvdmlkZXI+U0FLPC9Qcm92aWRlcj4NCjwvU291cmNlQ29udHJvbEluZm8+</span></</span>State</span>></span></span>
...</span></span></code></pre>decoded "state" contents:<</span>SourceControlInfo</span> xmlns</span>:</span>xsd</span>=</span>"</span>http://www.w3.org/2001/XMLSchema</span>"</span> xmlns</span>:</span>xsi</span>=</span>"</span>http://www.w3.org/2001/XMLSchema-instance</span>"</span> xmlns</span>:</span>ddl2</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2003/engine/2</span>"</span> xmlns</span>:</span>ddl2_2</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2003/engine/2/2</span>"</span> xmlns</span>:</span>ddl100_100</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2008/engine/100/100</span>"</span> xmlns</span>:</span>ddl200</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2010/engine/200</span>"</span> xmlns</span>:</span>ddl200_200</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2010/engine/200/200</span>"</span> xmlns</span>:</span>ddl300</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2011/engine/300</span>"</span> xmlns</span>:</span>ddl300_300</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2011/engine/300/300</span>"</span> xmlns</span>:</span>ddl400</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2012/engine/400</span>"</span> xmlns</span>:</span>ddl400_400</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2012/engine/400/400</span>"</span> xmlns</span>:</span>ddl500</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2013/engine/500</span>"</span> xmlns</span>:</span>ddl500_500</span>=</span>"</span>http://schemas.microsoft.com/analysisservices/2013/engine/500/500</span>"</span> xmlns</span>:</span>dwd</span>=</span>"</span>http://schemas.microsoft.com/DataWarehouse/Designer/1.0</span>"</span>></span></span>
  <</span>Enabled</span>></span>true</span></</span>Enabled</span>></span></span>
  <</span>ProjectName</span>></span>SAK</span></</span>ProjectName</span>></span></span>
  <</span>AuxPath</span>></span>SAK</span></</span>AuxPath</span>></span></span>
  <</span>LocalPath</span>></span>SAK</span></</span>LocalPath</span>></span></span>
  <</span>Provider</span>></span>SAK</span></</span>Provider</span>></span></span>
</</span>SourceControlInfo</span>></span></span></code></pre>Enough xml namespaces for you?</li>
The state base64 changes all the time causing diff noise.</li>
</ol>
I know I can't count, but if ssrs is going to make so little effort I don't see why I should. And to be honest "100" seemed a lot more like comic exaggeration when I titled my tomboy note which only had 5 or 6 grumblings in it, I wasn't actually expecting to get within spitting distance of the original number!</p>
Doesn't mean I won't use it again mind, just don't promise to like it.</p>
If you liked this, you might also like the ssrs deployment tool I sometimes look after: https://github.com/timabell/ssrs-powershell-deploy</a> (mostly not my work, just pulled together a bunch of contributions).</p>


LibreOffice mail merge - "data source 'SOURCE' was not found"
2015-01-12T22:31:00.002+00:00
So another year on, LibreOffice 4.2.7.2 (via Linux Mint 17.1) still has a dog's breakfast of a mail merge feature, hey ho, hopefully it might actually get fixed following the fork from OpenOffice and the change in contribution methods.</p>
Anyways</p>
So I've moved machines, copied my files across and for some reason my mail merge has soiled itself and now bleats "data source 'SOURCE' was not found"</em> which is as unhelpful as it is infuriating, especially given that the "check connections" button is exactly the wrong place to look for an answer.</p>
Turns out you actually get this if even just a single field in your document is 'broken'. How do you tell which ones are broken? Well you have to change them all to just be sure. Sigh.</p>
The fix for me today was as follows (though with such a messy feature there's unlimited ways it can break):</p>

Hit F4 and check that your connection to the spreadsheet actually exists and works, and unbreak anything you find therein. While you're in there you can marvel at how it requires a whole other file (.odb) just to remember how to get to a spreadsheet. (See below for fixing this)</li>
Turn on the field names so you can see what the f*** is actually going on with "View > Field Names (ctrl+f9)</em>" which will show you the fully qualified field name, which might even be completely wrong. You can now see that for whatever reason (insanity?) it embeds more than just the field name at the field place-holder.</li>
And finally the way you actually fix the broken fields it's failing to tell you about actually lies under the menu item "Edit > Fields</em>", where you can change all the broken references one at a time to the correct place.</li>
For bonus points, if it the field looks right but is silently broken somehow then you have to change the field to something else, hit okay, and then change it back again for anything to actually change, which is annoying if you have a lot of fields.</li>
</ol>
Fragile much?</p>
Another fix I've just discovered is you can rename your data source to match the name defined in the fields (assuming they're all the same) and it'll start working again.</p>
Fixing the .odb file</h4>
If you're stuck on point 1, here's how you fix it, also completely non-obvious and full of apparent dead-ends and dubious information.</p>

Give up on trying to do this in writer, it doesn't seem possible, in spite of false hope from the data sources tool, it only allows you to select .odb (database) files, not spreadsheets.</li>
Open up "libreoffice base", which pops open the database wizard</li>
Choose "connect to an existing database"</li>
In the dropdown choose "Spreadsheet"</li>
Next</li>
Browse for your spreadsheet</li>
Next</li>
Leave "register database for me" selected</li>
Leave "open the database for editing" checked</li>
Finish</li>
It prompts to save the new database (.odb), I suggest saving it in the same folder as the spreadsheet to save future confusion.</li>
You now have the database open in "base", you should see your spreadsheet sheets listed as tables</li>
Open a table (i.e. a sheet) and check you can see the spreadsheet contents</li>
Close "base", saving changes</li>
Return to your writer document</li>
Open the data sources again (F4), you should now be able to browse your spreadsheet via your newly created database.</li>
</ol>
Simpler than getting planning permission out of a vogon. :-/</p>
Hope that helps some other poor open source die-hard who has work to do.</p>
Useful refs:</p>

http://davidmburke.com/2011/08/10/mail-merge-in-libreoffice/</a></li>
https://forum.openoffice.org/en/forum/viewtopic.php?f=30&t=29708</a></li>
https://www.libreoffice.org/bugzilla/buglist.cgi?quicksearch=mailmerge</a></li>
http://ask.libreoffice.org/en/question/19590/can-you-import-a-libreoffice-calc-spreadsheet-into-a-libreoffice-database/</a></li>
</ul>


throw vs throw ex vs wrap and throw in c-sharp
2014-05-10T10:23:00+00:00
I've come across the throw</code> vs throw ex</code> 'debate' a few times, even as an
interview question, and it's always bugged me because it's never something I've
worried about in my own c# code.</p>
Background</h1>
So here's a typical example of the throw vs throw ex thing:
https://stackoverflow.com/questions/730250/is-there-a-difference-between-throw-and-throw-ex</a></p>
Basically it revolves around either messing up the line numbers in your stack
trace (throw ex;</code>) or losing a chunk of your stack entirely (throw;</code>) -
exception1 and 2 respectively in this nice clear answer:
http://stackoverflow.com/a/776756/10245</a></p>

</a>
</div>
The third option</h1>
I've just figured out why it's never been an issue for me.</p>
Because in my own code, whenever I catch and re-throw I always</em> wrap another
exception to add more context before rethrowing, and this means you don't have
either of the above problems. For example:</p>
private static void ThrowException3() {</span></span>
    try {</span></span>
        DivByZero(); // line 43</span></span>
    } catch (Exception ex) {</span></span>
        throw new Exception("doh", ex); // line 45</span></span>
    }</span></span>
}</span></span></code></pre>
Exception 3:</p>
System.Exception: doh ---> System.DivideByZeroException: Division by zero</span></span>
  at puke.DivByZero () [0x00002] in /home/tim/repo/puker/puke.cs:51</span></span>
  at puke.ThrowException3 () [0x00000] in /home/tim/repo/puker/puke.cs:43</span></span>
  --- End of inner exception stack trace ---</span></span>
  at puke.ThrowException3 () [0x0000b] in /home/tim/repo/puker/puke.cs:45</span></span>
  at puke.Main (System.String[] args) [0x00040] in /home/tim/repo/puker/puke.cs:18</span></span></code></pre>
Obviously 'doh' would be something meaningful about the state of that function
ThrowException3()</code> in the real world.</p>
Full example with output at
https://gist.github.com/timabell/78610f588961bd0a0b95</a></p>
This makes life much easier when tracking down bugs / state problems later on.
Particularly if you string.Format()</code> the new message and add some useful state
info.</p>


Why publish open source when you are commercial?
2014-03-08T19:07:00+00:00
Why open source your commercial projects?</p>

Forces you to decouple them from other internal systems.</li>
Encourages thinking in terms of reusable modules, which is better for
internal reuse just as much as public reuse.</li>
Possibility of contributions to systems useful to your business by others.</li>
Easier reuse within your organisation (the public internet is a better
search and sharing system than any internal systems).</li>
Reputation advantages, the best coders often like to work in open and
forward-thinking companies, and having public shared code is a great sign
of such an organisation.</li>
</ul>
Do it early</p>

Preferably push your very first commit straight to github.</li>
Do it before it has a chance to be tightly coupled to internal systems,
otherwise you'll have to unpick it and it will be less decoupled from day
one, and inertia might mean that in spite of the best intentions you then
never publish it.</li>
You'll have it in mind that every commit is public from day one, avoiding
adding internal config etc and forcing you to factor it out into config
which is all round a good thing.</li>
Don't wait for your code to be perfect, there are compromises in all code
and sharing something imperfect is better than sharing nothing.</li>
</ul>

</a>
</div>
Worried about the brand?</p>

Commit under personal email addresses and push to personal github accounts.
You can always setup a corporate github account later when you are feeling
more confident.</li>
</ul>
Of course I'm not saying you should open source everything, for example your
core product's codebase should probably not go on github if you are a product
company!</p>
Be brave, be open.</em></p>
Props to Tom Loosemoore</a></p>


Bash command line editing cheat sheet
2014-02-10T18:33:00+00:00
https://www.gnu.org/software/bash/manual/bashref.html#Readline-Interaction</a></p>


ctrl-a/e</strong> start/end of line</p>
</li>

alt-f/b</strong> forward/back a word</p>
</li>

ctrl-w/alt-d</strong> delete to start/end of word</p>
</li>

alt-.</strong> add argument from previous command (repeat to cycle) - love this one</p>
</li>

ctrl-shift-_</strong> undo (i.e. ctrl-underscore)</p>
</li>

ctrl-y</strong> paste (yank) deleted text</p>

alt-y</strong> paste older deleted text instead</li>
</ul>
</li>

prefix with alt+digit</strong> (0-9) to do multiple, e.g. delete two words</p>
</li>

start with alt-minus</strong> to go backwards</p>
</li>
</ul>
Just a few notes I threw together for my own benefit. I finally got around to learning a bit more about editing commands on the Linux shell / terminal.</p>

</a>
</div>


Converting kml to gpx with python
2014-02-03T20:27:00+00:00
Today I wanted to geo-code some of my photos.</p>
I have an SLR digital camera (no gps of course), and an android phone. I
recorded a track with My
Tracks</a>
from google on the phone. (Not entirely recommended but works). I then fired up
digikam to run the geo-correlation and add lat-long to the exif of the files
only to discover digikam doesn't know how to read
kml</a>.
Fooey.</p>

</a>
</div>
I looked to gpsbabel, but it apparently can't handle this style of kml file, as
differentiated by the coordinates being in the following style of markup:</p>
<gx:Track></span></span>
<when>2014-01-25T18:00:13.955Z</when></span></span>
<gx:coord>-1.885348 50.769434</gx:coord></span></span>
<when>2014-01-25T18:00:14.565Z</when></span></span>
<gx:coord>-1.885193 50.769328 53.20000076293945</gx:coord></span></span>
<when>2014-01-25T18:00:58.566Z</when></span></span></code></pre>
So I wrote a python script to munge it into gpx shape:</p>

Tim Abell

Running dotnet microservices in Aspire without a project reference

Using Gitopolis to Manage Multiple Git Repositories

Day One: Setup</h2> She installs Gitopolis and points it at her working directory:</p>

Day Two: Payoff</h2> Instead of jumping between folders, she runs everything from the top level.</p> Pull all backend repos:</p>

Use Checklists & Templates For Onboarding New People

Trello</h3> The trello checklist feature is pretty front and center, just create a card and there's a checklist button right there.</p> </p>

Trello</h3> </p> </p>

1. Automated setup</h3> Make your git repos require nothing more than a clone and some kind of run script to have a fully functioning local dev setup (there are tons of ways of doing this, bash, docker, makefiles, dotnet-aspire, 12-factor etc. etc.)</p>

Mandatory reading for tech leaders 📚

Tech lead lessons learned

Why not WhatsApp?

Light relief through satire</h2> Man that was depressing/heavy/boring.</p>

The Hidden Cost of Overusing AI Coding Tools: Tech Debt in Sheep’s Clothing

Published articles</h3> "It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it." - Ivan Gekht, CEO, Gehtsoft</p>

How to upload kindle fire kids photos to amazon photos

Other upset kindle users</h2> I've been having a world of pain trying to get the kids' photos off their old tablet before sending it back, fortunately before it was completely broken. Apparently I'm not alone:</p>

Step 1 - enable parent viewing of photos</h2> This was already enabled for me, but I'm assured this is a required step</p>

Check access to kids photos from adult profile</h2> Go into the grown up profile</li> Open the gallery app</li>

2. Download them</h3> </p> </p>

NuGet libraries to avoid

The NuGet Hall of Former Glory</h2>

Podcast: using AI for coding

Why is reading bad code so painful?!

Viewing git history for merged feature branches

Break down</h3> Let's break down what how this works (oh no, I'm starting to write like gpt! halp!)</p>

Store merge commit ref</h4> Put the sha1 of the merge into a variable (because we need it more than once)</p>

Find last feature branch commit before merge</h4> This subcommand uses log to show the parent commit sha1s, then uses cut to find the second parent sha1, i.e. the top commit of the feature branch that was merged in.</p>

Git log invocation</h4> Standard git log formatting to get a nice graph, can be replaced with tig</code> for a prettier graph, which is what I've done for my graphs below.</p>

Viewing feature A</h3> Viewing feature A in isolation is not too hard, you can run a git log or tig on the last commit before it was merged in (7714120</code>),</p>

Over-eager first-parent filter</h3> If you ask for first-parent with the sha1 of main and the branch you are no longer seeing everything that could be on the feature branch.</p>

Don't use Azure CosmosDB

The RAID Log: Your Guide to Surviving Chaos, Politics, and Petty Office Wars

How to Survive Nigel</h4> Use his own vagueness against him. When he logs a risk with zero substance, reply with, “What mitigation steps do you suggest?” Watching Nigel scramble for an answer is its own reward.</p>

How to Outplay Betty</h4> Always specify your deliverables and timelines in your own RAID log entries. If Betty tries to blame you, you can counter with, “Actually, Betty, this was logged with an agreed deadline last week.” Serve, ace, point.</p>

How to Avoid Drowning in Eddie’s Flood</h4> Politely suggest a RAID review. This is where you “prioritize high-impact risks” and quietly delete “Potential email server outages due to Mercury retrograde.”</p>

How to Defuse Rick’s Bombs</h4> Ask for specifics the moment Rick logs a risk. “What exact tests showed performance concerns? Can we see the results?” Rick hates specifics almost as much as Nigel hates understanding tech.</p>

More Chaos on the Battlefield: How RAID Logs Go Off the Rails</h3>

Turning the Chaos to Your Advantage</h2>

More Ways to Use RAID Logs to Your Advantage</h3>

Good: Risk Triage Hero</h4> During chaotic RAID meetings, step up as the voice of reason. Say things like, “Let’s focus on the high-impact risks and save the smaller ones for offline discussion.” People will start associating you with clarity and efficiency.</p>

Evil: Risk Triage Saboteur</h4> Use triage as a power move. Suggest deprioritizing risks logged by people you secretly want to undermine (cough Betty cough). Frame it as “streamlining,” and nobody will notice you’re sabotaging their credibility.</p>

Good: The RAID Whisperer</h4> Offer to manage the RAID log for your team. Keep it accurate, concise, and aligned with project goals. You’ll gain trust and authority over the process, ensuring the log is actually useful.</p>

Good: Clarify Assumptions</h4> If assumptions are logged vaguely, clarify them in a way that benefits the team. Example: “Assume resource availability for Q4 testing.” If questioned later, you can point out that the assumption was clearly documented.</p>

Evil: Assumption Ambiguity</h4> Log assumptions so broad that they’re impossible to disprove. Example: “Assume smooth transition to production.” If anything goes wrong, shrug and say, “Well, we assumed everything would go right.”</p>

Trello</h3>
The trello checklist feature is pretty front and center, just create a card and there's a checklist button right there.</p>
</p>

Trello</h3>
</p>
</p>

2. Download them</h3>
</p>
</p>

Break down</h3>
Let's break down what how this works (oh no, I'm starting to write like gpt! halp!)</p>

How to Survive Nigel</h4>
Use his own vagueness against him. When he logs a risk with zero substance, reply with, “What mitigation steps do you suggest?” Watching Nigel scramble for an answer is its own reward.</p>

How to Outplay Betty</h4>
Always specify your deliverables and timelines in your own RAID log entries. If Betty tries to blame you, you can counter with, “Actually, Betty, this was logged with an agreed deadline last week.” Serve, ace, point.</p>

How to Avoid Drowning in Eddie’s Flood</h4>
Politely suggest a RAID review. This is where you “prioritize high-impact risks” and quietly delete “Potential email server outages due to Mercury retrograde.”</p>

How to Defuse Rick’s Bombs</h4>
Ask for specifics the moment Rick logs a risk. “What exact tests showed performance concerns? Can we see the results?” Rick hates specifics almost as much as Nigel hates understanding tech.</p>

Good: Risk Triage Hero</h4>
During chaotic RAID meetings, step up as the voice of reason. Say things like, “Let’s focus on the high-impact risks and save the smaller ones for offline discussion.” People will start associating you with clarity and efficiency.</p>

Evil: Risk Triage Saboteur</h4>
Use triage as a power move. Suggest deprioritizing risks logged by people you secretly want to undermine (cough Betty cough). Frame it as “streamlining,” and nobody will notice you’re sabotaging their credibility.</p>

Good: The RAID Whisperer</h4>
Offer to manage the RAID log for your team. Keep it accurate, concise, and aligned with project goals. You’ll gain trust and authority over the process, ensuring the log is actually useful.</p>

Good: Clarify Assumptions</h4>
If assumptions are logged vaguely, clarify them in a way that benefits the team. Example: “Assume resource availability for Q4 testing.” If questioned later, you can point out that the assumption was clearly documented.</p>

Evil: Assumption Ambiguity</h4>
Log assumptions so broad that they’re impossible to disprove. Example: “Assume smooth transition to production.” If anything goes wrong, shrug and say, “Well, we assumed everything would go right.”</p>